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ABSTRACT 


The  purpose  of  this  thesis  is  to  analyze  the  application  of  Modeling  and  Simulation 
(M&S)  within  the  Army  Operational  Test  and  Evaluation  (OT&E)  process  in  support  of 
weapon  systems  acquisition.  This  thesis  considers  the  Army’s  current  acquisition  process, 
M&S  technologies,  infrastructure,  and  policies  that  guide  the  Program  Manager  (PM)  in  the 
application  of  modeling  and  simulation  in  operational  testing.  An  analysis  of  the  potential 
strengths  and  weaknesses  of  M&S  in  addressing  OT&E  issues  is  presented.  Lessons  learned 
from  past  OT&E  efforts  are  analyzed  for  process  improvement  through  M&S  applications. 
The  analysis  indicates  that  M&S  is  a  viable  tool  for  assisting  the  PM  in  completing  OT&E. 
M&S  techniques  can  assist  in  test  design  validation,  expand  testing  in  areas  of  limited 
resources  and  environmental  concerns,  and  validate  live  testing  data.  From  this  analysis,  a 
set  of  recommendations  are  formulated,  indicating  where  the  PM  can  integrate  M&S  into  the. 
OT&E  process. 
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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  the  application  of  Modeling  and  Simulation 
(M&S)  within  the  Army  Operational  Test  and  Evaluation  (OT&E)  process  in  support  of  weapon 
systems  acquisition.  The  focus  is  to  provide  Program  Managers  (PMs)  a  viable  tool  in  managing 
the  Army  OT&E  process.  From  this  analysis,  a  set  of  recommendations  in  the  area  of  M&S 
integration  into  Army  OT&E  are  formulated  and  offered.  The  recommendations  address  both  the 
cost  effectiveness  and  the  adequacy  of  the  application  of  M&S  within  the  Army  OT&E  process. 

B.  BACKGROUND 

The  modem  battlefield  requires  continuous  advancement  of  technology  in  virtually  all 
complex  weapon  systems.  These  complex  systems  provide  accuracy,  reduce  manpower 
requirements,  and  create  the  ability  to  control  vast  numbers  of  tasks  simultaneously.  However, 
because  of  their  complexity,  they  pose  a  challenge  to  the  OT&E  process  due  to  the  high  cost  and 
difficulty  of  testing  them  in  an  operational  environment.  These  difficulties  arise  from  the 
complex  nature  of  weapons  on  the  modem  battlefield  and  the  need  to  maintain  “technologically 
current”  weapon  systems.  Two  examples  include  weapons  tracking  systems  capable  of 
identifying  and  tracking  multiple  targets  and  communications  systems  capable  of  simultaneously 
processing  thousands  of  analog  and  digital  messages. 

Within  the  Army  OT&E  process  of  weapon  systems  development,  M&S  is  an  emerging 
technology  that  is  available  for  application  in  the  defense  acquisition  process.  Specifically, 
within  the  Army  OT&E  process,  this  technology  offers  great  promise  to  decrease  the  amount  of 
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time  and  money  spent.  As  the  defense  budget  continues  to  rapidly  decline,  technologies 
comprising  revolutionaiy'  techniques  in  M&S  are  receiving  increasingly  wider  interest  and 
application  within  the  Army  OT&E  process.  The  use  of  M«&S  in  the  Army  OT&E  process  of 
w'eapon  systems  acquisition  can  help  the  PM  analyze  and  identify  potential  areas  of  concern 
before  significant  funds  are  spent  in  live  testing  and  advanced  development.  Early  use  of  M&S 
in  the  Army  OT&E  process  can  provide  unique  insight  into  system  problems.  As  the  cost  of 
OT&E  continues  to  increase  with  the  complexity  of  today’s  weapon  systems,  the  application  of 
M&S  in  the  Army  OT&E  process  becomes  ever  more  appealing. 

Counterbalancing  the  appeal  of  the  relatively  low  cost  of  M&S  for  OT&E  applications 
are  laws,  directives,  regulations,  and  policies  mandating  live  testing.  In  the  test  and  evaluation 
community,  there  is  a  widely  held  distrust  for  anything  that  is  “simulated,”  including  simulation 
used  in  OT&E.  The  validity  of  the  simulation  model  is  a  primary  concern  in  developing  viable 
information  and  reliable  test  data  for  the  PM.  A  critical  issue  that  must  be  considered  in 
developing  the  use  of  M&S  in  the  Army  OT&E  process  is  the  suitability  of  the  simulation  model 
in  replicating  the  real-world  environment.  Simulation  applications  have  the  potential  to  evaluate 
virtual!)  eveiy'  component  of  a  weapon  system,  in  all  environments,  providing  large  amounts  of 
data  to  assist  the  PM  in  managing  weapons  system  acquisition.  Accredited  M&S  techniques, 
used  in  the  Army  OT&E  process,  have  the  potential  to  assist  the  PM  in  delivering  combat-ready 
weapon  systems  at  significant  reductions  in  both  cost  and  time  to  deliver)'. 

C.  THESIS  OBJECTIVE 

This  thesis  anaty  zes  the  application  of  M&S  in  the  Army  OT&E  process  in  support  of 
weapon  systems  acquisition.  The  current  use  of  M&S  in  the  Army’s  OT&E  process  is  analyzed 


2 


to  identify  persistent  problems  and  issues,  and  assess  shortcomings  in  its  application.  Analysis  is 
offered  where  M&S  concepts  have  potential  application  for  enhancing  the  Army  OT&E  process 
as  to  the  type,  extent,  and  applicability  of  M«&:S  concepts. 

Analysis  of  the  potential  strengths  and  weaknesses  of  modeling  and  simulation  in 
addressing  the  elements  and  issues  associated  with  the  Army  OT«&E  process  are  presented. 
Lessons  learned  from  past  OT&E  efforts  are  analyzed  for  possible  process  improvement  through 
M&S  integration.  Supporting  conclusions  and  recommendations  based  on  the  analysis  of  the 
thesis  are  presented.  Areas  for  further  research  are  identified  to  focus  future  study. 

D.  RESEARCH  QUESTIONS 

The  primary  research  question  for  this  thesis  is: 

1 .  What  current  role  does  M&S  play  and  what  future  role  should  it  play  in  support 
of  the  Army  OT&E  process? 

The  subsidiary  research  questions  are: 

1.  How  does  M&S  support  the  Army  OT&E  process? 

2.  What  regulatory  and  policy  guidelines  are  established  for  governing  the  use  of  M&S 
in  the  Army  OT&E  process? 

3.  What  types  of  M&S  are  currently  being  used  within  the  Army  OT&E  process? 

4.  What  types  of  Army  OT&E  methods  lend  themselves  to  the  use  of  M&S? 

5.  What  are  the  key  elements  to  consider  when  using  M&S  in  the  Army  OT&E 
process? 

6.  What  are  the  capabilities  of  M&S  in  relationship  to  the  Army  OT&E  process? 

7.  What  type  of  information  can  be  gained  from  using  M&S? 


E. 


SCOPE  AND  LIMITATIONS 


The  thrust  of  this  thesis  concerns  M&S  applications  within  the  Army  OT&E  process  in 
support  of  weapon  systems  acquisition.  The  potential  issues  associated  with  the  application  of 
M&S  in  this  thesis  are  limited  to  the  OT&E  process.  Specifically,  this  thesis  addresses  the 
OT&E  process  rather  than  the  Developmental  Test  and  Evaluation  (DT&E)  process. 

This  thesis  identifies  potential  opportunities  to  integrate  M&S  in  the  Army  OT&E 
process.  The  development  of  a  common  foundation  of  knowledge  that  governs  the  Army  OT&E 
process  will  establish  the  various  applications  of  M&S  as  an  integral  part  of  OT&E.  The  laws, 
regulations,  directives,  and  policies  pertaining  to  Army  OT&E  are  examined  to  identify  where 
and  under  what  circumstances  M&S  integration  can  be  accomplished  and  where  it  is  prohibited. 
Next,  a  basic  understanding  of  M&S  techniques  and  applications  will  establish  the  plausibility  of 
using  M&S  in  the  Army  OT&E  process.  Finally,  current  Army  acquisition  programs  are 
analyzed  to  e\'aluate  specific  successes  and  failures  when  integrating  M&S  into  the  Army  OT&E 
process. 

F.  RESEARCH  LITERATURE  AND  METHODOLOGY 

Research  data  were  obtained  from  official  Government  directives  and  policies,  journals, 
previous  theses.  United  States  Code,  DoD  and  Army  regulations  and  manuals,  and  personal 
inters  lews.  Information  on  current  M&S  programs  was  obtained  from  Army  Training  and 
Doctrine  Command  (TRADOC),  Analysis  Centers  (TRACs),  the  Army’s  Simulation,  Training 
and  Instrumentation  Command  (STRICOM),  and  the  Defense  Modeling  and  Simulation  Office 
(DMSO).  Information  on  the  U.S.  Army  Operational  Test  and  Evaluation  process  was  obtained 
from  the  Operational  Test  and  Evaluation  Command  (OPTEC)  and  the  Test  and  Experimentation 


Command  (TEXCOM).  Current  directives  and  policies  guiding  the  Army  OT&E  process  were 
reviewed.  A  comprehensive  literatme  search  was  conducted  to  determine  the  existence  and 
validity  of  available  simulation  models  and  techniques. 

Research  was  conducted  via  personal  and  telephone  interviews  with  cognizant  M&S  and 
Test  and  Evaluation  (T&E)  personnel.  Interviews  with  OT&E  related  individuals  concentrated 
on  the  opportunities  for  M&S  to  enhance  the  OT&E  process.  Interviews  with  M&S-related 
individuals  focused  on  the  state-of-the-art  in  M&S  and  its  applicability  to  the  Army  OT&E 
process. 

G.  ACRONYMS 

An  extensive  listing  of  acronyms  associated  with  both  OT&E  and  M&S  subjects  is 
presented  in  the  Appendix. 

H.  ORGANIZATION  OF  THESIS 

Chapter  I  contains  the  purpose,  backgroimd,  thesis  objective,  research  questions,  scope 
and  limitations,  methodology,  and  thesis  organization. 

Chapter  II  of  this  thesis  addresses  the  acquisition  process  and  how  the  Army  OT&E 
process  supports  new  weapon  systems  acquisition.  It  explores  the  relationship  between  the  Army 
DT&E  process  and  OT&E  process,  establishing  the  interface  between  M&S  and  the  Army 
OT&E  process. 

Chapter  III  explores  the  basics  of  M&S  as  a  tool  for  assisting  the  PM.  It  establishes  a 
baseline  of  information  about  M&S  including  key  elements  in  simulation,  types  of  simulation 
available,  capabilities  of  simulation,  and  types  of  information  which  simulation  provides  the 
PM. 
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Chapter  IV  outlines  the  Army’s  current  guidelines  and  policies  that  affect  the  use  of  M&S 
in  the  Army  OT&E  process.  It  then  compares  and  contrasts  Army  directives  v^dth  those 
established  by  DoD.  Current  facilities  that  support  M«feS  are  investigated  to  assist  the  PM  in 
using  simulation  in  the  Army  OT&E  process. 

Chapters  V  and  VI  contain  analysis  of  M&S  applications  in  actual  weapon  systems  and 
provides  recommendations  for  future  applications  of  M&S  in  the  Army  OT&E  process.  Areas 
for  further  research  are  also  identified. 
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II.  THE  ACQUISITION  PROCESS  AND  OT&E 


A.  GENERAL 

The  United  States  and  the  Army  are  facing  a  dynamic  international  environment  which 
involves  a  complex  array  of  security  issues,  xmpredictable  threats  and  a  fiscally  restrained 
domestic  budget.  In  this  environment,  the  Army  needs  to  draw  on  its  defense  acquisition 
professionals  to  help  overcome  formidable  challenges  of  rapidly  fielding  technically  superior 
battlefield  weapon  systems.  In  an  address  to  the  National  Contract  Management  Association  in 
1993,  Dr.  William  J.  Perry,  as  Deputy  Secretary  of  Defense,  explained  the  importance  of  fielding 
technologically  superior  weapon  systems.  He  stated,  “The  advantage  we  had  in  Desert  Storm 
had  three  components.  We  had  an  advantage  in  people,  an  advantage  in  readiness,  and  an 
advantage  in  technology.”  (Piplani,  1994,  p.  v)  The  technological  advantage  achieved  by  the 
military  is  maintained  through  the  acquisition  process.  The  acquisition  process  is  the  primary 
means  through  which  the  Army  procures  new  equipment  and  technology  to  maintain  the 
“technology  advantage.” 

Operational  Test  and  Evaluation  (OT«feE)  is  a  critical  component  of  the  acquisition 
process  that  ensures  weapon  systems  are  fielded  in  a  combat-ready  status.  As  part  of  the 
acquisition  process,  OT&E  is  designed  to  assess  a  system's  operational  effectiveness  and 
suitability  in  a  realistic  operational  environment.  The  most  capable  system  is  useless  if  it  cannot 
be  employed  by  the  intended  user  within  doctrinal,  tactical,  force  structure,  and  training 
requirements. 
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Modeling  and  Simulation  (M&S)  offers  a  new  technology  that  will  streamline  the  time 
and  funding  spent  in  OT&E.  Simulation  and  modeling  techniques  can  be  applied  to  every  major 
Army  weapon  development  program  to  reduce  design  and  production  cost,  improve  performance, 
improve  diagnostics  and  maintenance,  assist  in  better  and  faster  training  of  personnel,  and 
improve  command  and  control  on  the  battlefield.  (Piplani,  1994,  p.  v) 

The  purpose  of  this  chapter  is  to  briefly  review  the  Army  acquisition  process  and  delineate 
the  role  of  OT«S:E  within  the  acquisition  process.  The  chapter  will  conclude  with  the  integration 
of  M&S  in  support  of  OT&E. 

B.  THE  ACQUISITION  PROCESS 

The  roots  of  the  current  defense  systems  acquisition  process  can  be  traced  to  the  early 
1950s  Over  the  years  many  reform  initiatives  have  been  implemented  to  make  the  acquisition 
process  more  efficient  and  effective.  However,  the  goal  of  the  acquisition  process  has  remained 
constant  throughout  the  years,  “.  .  .  to  procure  state-of-the-art  technology  and  products,  rapidly, 
from  reliable  suppliers  who  utilize  the  latest  manufacturing  and  management  techniques  that 
satisfies  a  specific  user’s  need  at  an  affordable  cost.”  (Piplani,  1994,  p  1-4)  The  policy 
foundation  for  the  acquisition  process  is  found  in  the  DoD  5000  series. 

The  DoD  5000. 1  establishes  broad  policies  governing  the  defense  systems  acquisition 
process  It  states  that  the  three  decision-making  support  systems  must  interact  and  interface  with 
each  other  in  order  for  the  process  to  work  effectively.  The  three  systems  illustrated  in  Figure  1 
are:  1)  requirements  generation,  2)  acquisition  management  and  3)  planning,  programming  and 
budgeting  (PPBS).  This  triad  of  effort  is  supported  by  a  structured  program  of  checks  and 
balances.  Figure  2  depicts  the  milestones  in  the  acquisition  process.  Within  the  structure  of  the 
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acquisition  process,  test  and  evaluation  verifies  the  maturity  of  system  design  and  fimctionalily  of 
system  work  performed  during  each  milestone. 


Start 

Phase  0 

Phase  I 

Phase  II 

Phase  III 

Preliminary 

TEMP 

Update 

TEMP 

Update 

TEMP 

Developmental  Test  and  Evaluation 
DT^.E 


Operational  Test  and  Evaluation 
OT&E 


Figure  2.  The  Acquisition  Process  Milestone  Support  Structure  (Hoivik,  1996) 
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C-  OT&E  SUPPORT  OF  THE  ACQUISITION  PROCESS 

The  acquisition  process  can  take  many  years  from  the  time  a  weapon  system  requirement 
is  identified  until  the  weapon  system  is  fielded.  As  a  weapon  system  progresses  through  the 
acquisition  process,  test  and  evaluation  emphasis  turns  from  Developmental  Test  and  Evaluation 
(DT&E)  toward  Operational  Test  and  Evaluation  (OT«feE).  The  fundamental  purpose  of  OT&E  in 
the  acquisition  process  is  to  demonstrate  the  feasibility  of  the  conceptual  approach,  reduce  design 
risk,  identify  design  alternatives,  and  estimate  operational  effectiveness  and  suitability.  OT&E 
results  w'eigh  heavily  on  the  decisions  to  continue  development,  to  accept  the  system,  or  to 
change  organization,  doctrine,  and  concepts  that  define  the  weapon  system  requirements.  (DSMC 
TEMG,  1988,p.l-l)  Figure  2  depicts  the  Army  T&E  events  in  support  of  the  acquisition 
process. 

OT&E  has  an  Independent  Agency  responsibility  designed  to  determine  if  the  weapon 
system  meets  user  requirements.  OT&E  verifies  the  effectiveness  and  suitability  of  the  weapon 
system.  The  T&E  orgamzation,  depicted  in  Figure  3,  illustrates  its  independent  nature  and 
authority  channel.  As  the  weapon  system  nears  the  end  of  the  acquisition  process,  there  is 
tremendous  political  and  bureaucratic  pressure  to  begin  full-scale  production.  The  operational 
testers  are  often  viewed  as  “show  stoppers"  because  successful  completion  of  OT&E  is  required 
before  an}-  weapon  system  can  progress  beyond  low  rate  initial  production  (LRIP).  (LeSueur, 
1994,  p.l2) 

D.  THE  OT&E  ROLE 

OT&E  programs  are  structured  to  determine  the  operational  effectiveness  and  suitability- 
of  a  s}  stem  under  realistic  conditions  (e.g.,  combat)  and  to  decide  if  the  minimum  acceptable 
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operational  performance  requirements  as  specified  in  the  Operational  Requirements  Document 
(ORD)  have  been  satisfied.  (DoD  5000.2-R,  1996,  p.l7) 

Operational  performance  requirements  refer  to  the  weapon  system's  combat  effectiveness 
achieved  when  operated  and  maintained  by  typical  users,  within  the  intended  doctrine  and  tactics, 
and  as  part  of  standard  organizations  integrated  with  other  battlefield  operating  systems. 
Achieved  operational  effectiveness  and  suitability  are  distinct  from  weapon  system  capabilities 
(e.g.,  speed,  range,  penetration,  etc.)  determined  during  DT&E. 

Realism  is  a  critical  element  of  the  OT&E  process.  Realistic  combat  conditions  are  those 
that  represent  the  environment,  doctrine,  level  of  training,  and  structure  in  which  the  evaluated 
system  would  normally  be  expected  to  operate  within  a  combat  environment. 


Operational  Test  and 
Evaluation  Command 
(OPTEC) 


Operational  Evaluation 
Co;nmand  (OEC) 

OPERATIONAL 

EVALUATION 


Test  and  Experimentation 
Command  (TEXCOM) 

OPERATIONAL 

TESTING 


"Honest  Bro^rer" 


"Rigorous  &  Stresmiined" 


Figure  3.  T&E  Organization  Structure  (Burt,  1996) 
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E.  THE  OT&E  ORGANIZATION 

Congress  passed  legislation  creating  the  Office  of  Operational  Test  and  Evaluation  in 
1983  with  the  mission  to  independently  evaluate  the  Services'  weapon  systems  tests  and  assess 
test  results.  As  part  of  that  legislation,  the  Director  of  Operational  Test  and  Evaluation 
(DOT&E)  reports  directly  to  the  Secretary  of  Defense  and  is  given  broad  authority  to  suspend 
major  weapons  programs  that  perform  poorly  in  operational  tests.  In  addition  to  reporting  to  the 
Secretary  of  Defense,  the  DOT&E  reports  directly  to  both  the  House  and  the  Senate  Armed 
Services  Committees,  at  least  annually,  regarding  OT&E  of  weapon  systems  being  developed. 

The  goal  of  the  legislation  is  to  save  time  and  money  by  exposing  problems  before  expensive 
weapons  are  purchased  and  fielded.  The  T&E  organization  is  depicted  in  Figure  3. 

The  Operational  Test  and  Evaluation  Command  (OPTEC)  is  the  Army's  OT&E 
organization  and  was  established  15  November  1990  by  Secretary  of  the  Army  General  Order 
Number  6.  The  organization,  depicted  in  Figure  4,  consists  of  the  OPTEC  Headquarters  and 
Support  Agencies,  the  Operational  Evaluation  Command  (OEC)  and  the  Test  and 
Experimentation  Command  (TEXCOM).  OPTEC's  mission  is  to  conduct  and  monitor  user  test 
and  evaluation  (except  medical)  for  the  Army.  OPTEC's  test  and  evaluation  include  Initial 
Operational  Test  and  Evaluation  and  Follow-on  Operational  Test  and  Evaluation  (lOTE  and 
FOTE)  in  support  of  the  acquisition  process,  force  development  testing  and  experimentation 
(FDTE),  concept  evaluation  program  (CEP)  trials,  early  user  test  and  evaluation  (EUTE),  and  the 
Army  part  of  joint  test  and  evaluation  (JT&E). 

OPTEC  is  a  field  operating  agency  of  the  Office  of  the  Chief  of  Staff  of  the  Army.  In 
keeping  with  the  Defense  Directives,  OPTEC  reports  the  results  of  Army  OT&E  through  the 
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Vice  Chief  of  Staff  of  the  Army  directly  to  the  Army  Chief  of  Staff.  The  main  part  of  OPTEC's 
mission  is  the  planning,  conducting,  and  reporting  of  Army  OT&E  required  hy  law  since  1972. 
Additionally,  OPTEC  conducts  tests  for  Training  and  Doctrine  Command  (TRADOC)  in 
support  of  its  mission  to  develop  combat  doctrine,  organizations,  and  material  requirements. 

F.  OT&E  IN  EACH  PHASE  OF  THE  ACQUISITION  PROCESS 

During  the  Concept  Exploration  Phase  (Phase  Zero)  prior  to  Milestone  One,  laboratory 
T&E  is  used  to  validate,  demonstrate  and  assess  the  capabiUties  of  key  subsystems  and 
components.  Studies,  analysis,  simulation,  and  test  data  are  used  to  erqrlore  and  evaluate 


Figure  4.  OPTEC  Organizational  Structure  (OPTEC  Homepage,  1996) 
alternative  concept  designs  proposed  to  satisfy  user-generated  requirements.  OT&E  conducted 
during  this  phase,  called  early  operational  assessment,  investigates  deficiencies  identified  during 
the  initial  mission  area  analysis.  OPTEC  monitors  concept  exploration  for  future  T&E  planning 
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and  provides  effectiveness  and  suitability  information  needed  by  the  program  manager. 
Operational  assessments  that  address  the  operational  impact  of  a  weapon  system’s  technical 
approach  are  conducted. 

During  the  Program  Definition  and  Risk  Reduction  (PDRR)  Phase  (Phase  One), 
operational  effectiveness  and  suitability  assessments  are  conducted.  Information  on  tactics, 
doctrine,  personnel  requirements,  and  organizational  impacts  of  the  weapon  system  are  gathered. 
OT&E  assessments  are  used  to  support  the  Milestone  II  decisions  for  developing  promising 
alternatives  for  entering  Phase  II,  Engineering  and  Manufacturing  Development  (EMD). 

The  objective  of  the  Engineering  and  Manufacturing  Development  Phase  (Phase  Two)  is 
to  design,  fabricate,  and  test  systems  that  closely  approximate  the  final  product.  Prior  to  the 
Milestone  III  decision,  a  dedicated  lOTE  is  conducted  on  the  weapon  system.  This  formally 
certifies  the  weapon  system  for  “final  OT&E.”  OT&E  has  the  greatest  impact  on  major 
programs  during  this  phase  because  the  decision  to  proceed  beyond  TRIP  is  contingent  upon 
successfully  completing  the  lOTE.  A  formal  final  Operational  Evaluation  (OPEVAL)  is 
required  prior  to  a  Milestone  111  decision.  The  specific  role  of  OT&E  during  Phase  Two  is  to: 

1 .  Estimate  the  operational  effectiveness  and  suitability  of  the  system. 

2.  Identify  operational  deficiencies. 

3.  Recommend  and  evaluate  changes  in  production  configurations. 

4.  Provide  information  for  developing  and  refining  logistics  support  requirements. 

5.  Estimate  the  surs'ivability  of  the  system  in  the  operational  environment. 


■*  A 
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Also,  during  post-production  OT&E  includes  FOTE  programs  designed  to  verify  the 
operational  effectiveness  and  suitability  of  the  production  system  and  to  determine  if  deficiencies 
identified  during  lOTE  have  been  corrected.  FOTEs  also  refine  doctrine,  tactics,  techniques,  and 
training  programs  for  the  life  of  the  system.  (DSMC  TEMG,  1988) 

G.  THE  CRITICAL  ELEMENTS  OF  OT&E 

The  OT&E  process  is  guided  by  operationally-related  test  and  evaluation  elements.  These 
elements  are  unique  to  the  OT&E  process  and  drive  the  design  of  all  OT&E  programs.  Because 
OT&E  is  designed  to  test  weapon  systems  imder  "combat"  conditions,  these  key  elements  are 
critical  to  effective  testing.  (OPTEC  73-21-1, 193) 

1.  Realism 

Realistic  environments  are  essential  to  achieving  the  goals  of  OT&E.  A  realistic 
environment  is  one  that  represents  the  conditions,  doctrine,  level  of  training,  and  structure  in 
which  the  evaluated  system  would  normally  be  expected  to  operate  in  a  combat  environment. 
These  environments  are  distinctly  different  from  those  encoimtered  in  DT&E  where  the 
environment  is  carefully  controlled. 

Realism  in  the  OT&E  test  process  is  affected  by  the  resources  available  to  replicate  the 
representative  threat  and  fiiendly  force  array,  create  the  desired  combat  battlefield  environment, 
and  evaluate  the  system  over  time.  Other  factors  affecting  realism  include  the  degree  to  which 
test  participants  represent  typical  operational  personnel,  test  personnel  familiarity  with  test  ranges 
and  maneuver  areas,  and  limitations  created  by  safety  or  environmental  concerns. 

The  conditions  present  during  OT&E  must  reach  the  limits  of  the  evaluated  system 
within  the  doctrinal  envelope  of  operation.  Stimulators,  external  data  provided  to  test  the  system, 
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are  used  to  stress  the  limits  of  the  evaluated  system.  An  example  of  such  stimulators  would  be 
the  external  data  used  to  stress  communications  and  software-intensive  components  of  an 
evaluated  system  to  levels  expected  in  a  sustained  combat  environment.  Effective  OT&E  of  a 
system  also  includes  testing  its  interoperability  characteristics.  This  includes  creating  a  realistic 
structure  of  associated  battlefield  operating  systems  to  evaluate  interoperability. 

2.  User  Orientation 

An  integral  member  of  the  OT&E  process  is  the  "user."  Representative  personnel  having 
the  correct  organizational  grade  and  Military  Occupational  Skill  (MOS),  level  of  training, 
experience,  and  aptitude  are  required  for  an  unbiased  operational  evaluation  of  the  system. 
"Users"  include  persormel  that  operate,  maintain,  support,  or  provide  command  and  control 
functions  affecting  the  evaluated  system.  The  term  “user"  specifically  excludes  contractor  and 
other  personnel  who  would  not  normally  be  involved  in  the  operation,  control,  or  maintenance  of 
the  ev  aluated  system.  To  avoid  bias,  it  is  important  that  the  personnel  selected  to  represent  the 
user  are  indeed  representative. 

3.  Representative  Systems 

The  evaluated  system  equipment  must  be  sufficiently  mature  to  be  considered 
functionally  representative  of  the  versions  eventually  fielded.  Software  must  be  complete  and  as 
near  to  the  fielded  version  as  possible  when  OT&E  is  conducted.  All  associated  equipment, 
publications,  training  programs,  and  Test  Measurement  Diagnostic  Equipment  (TMDE)  should 
be  complete  before  the  evaluation.  The  evaluated  system  is  usually  part  of  a  larger  table  of 
organization  and  equipment  and  interacts  or  is  supported  by  other  standard  systems.  These 
systems  must  also  be  representative  of  those  found  in  an  operational  environment. 
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4.  Sufficiency 

Sufficiency  in  the  OT&E  process  ensures  that  the  test  plan  addresses  all  of  the  issues 
specified  in  the  Operational  Requirements  Document  (ORD).  The  data  collection  plan 
establishes  the  means  for  collecting  sufficient  data  and  the  evaluation  of  the  collected  data.  The 
structure  and  use  of  OT&E  methods  for  acquiring  usable  information  are  critical  for  a  complete 
and  imbiased  evaluation. 

The  test  plan  is  the  key  document  in  determining  what  data  and  imder  what  circumstances 
data  are  collected.  The  test  plan  is  driven  by  the  user  provided  issues  specified  in  the  ORD. 

These  user  issues,  termed  Critical  Operational  Issues  (COI),  drive  the  test  plan  to  ensure  all 
requirements  for  evaluation  are  covered  in  testing.  Sufficient,  unbiased  iterations  of  test  events 
must  be  accumulated  to  facilitate  the  evaluation  techniques  prescribed  in  the  evaluation  plan. 

H.  OT&E  AND  M&S  INTEGRATION 

Modeling  and  Simulation  (M&S)  in  support  of  OT&E  is  an  emerging  application  field. 
M&S  supports  the  OT&E  process  of  weapon  systems  development  as  they  proceed  through  the 
acquisition  life  cycle.  Figure  5  depicts  M&S  application  within  the  acquisition  life  cycle. 
Currently,  the  Army  uses  a  wide  range  of  M&S  methods  in  the  OT&E  process.  These  include 
engineering  models,  battlefield  environment  models,  and  combat  models.  M&S  can  assist 
program  managers  by  reducing  OT&E  testing  costs.  Areas  for  particular  savings  include:  test 
scenario  development,  operational  analysis  of  weapon  systems,  sequencing  of  test  events, 
refinement  of  testing  objectives,  and  development  of  measures  of  effectiveness.  Additionally, 
M&S  can  include  identifying  parameters  and  drivers  for  field  tests.  M&S  can  assist  in 
determining  high  risk  areas,  predicting  test  results,  and  assisting  in  testing  due  to  safety  and 


environmental  constraints. 


The  primary  goal  of  OT&E  is  operational  evaluation.  Many  OT&E  methods,  including 
live  and  simulated  testing,  are  used  to  reach  that  goal.  M&S  can  replicate  each  of  the  four 
critical  elements  of  OT&E,  supporting  complete  operational  evaluation.  Because  live  testing 
carries  a  significant  cost  burden,  many  opportunities  are  lost  to  gather  valuable  data  of  some 
weapon  systems  over  the  life  cycle.  In  many  weapon  systems,  repetitive  testing  is  critical  in 
gaining  the  information  needed  to  completely  evaluate  the  system.  M&S  provides  the  ability  to 
conducted  repetitive  testing  over  time  at  reduced  cost.  If  OT&E  were  limited  to  live  testing,  it 
\s  ould  lose  much  of  its  opportunity  to  contribute  to  decisions  on  whether  to  produce  a  system 
based  on  extended  use.  Life  cycle  costing  is  critical  and  without  including  simulation  in  OT&E, 
knowledge  of  the  life  cycle  cost  would  be  reduced.  (Mann,  1983,  p.  6) 


Figure  5.  M&S  Integration  in  the  Acquisition  Life  Cycle  (OPTEC  Homepage,  1996) 
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OPTEC  recognizes  the  use  of  M&S  to  achieve  adequate  realism,  support  economical  test 
execution,  and  provide  complete  and  adequate  evaluations.  However,  to  ensure  that  OPTEC 
rnaintains  its  objective  as  an  independent  evaluator,  every  M&S  application  used  in  the  OT&E 
process  must  be  subjected  to  Verification,  Validation,  and  Accreditation  (W&A).  W&A 
ensues  that  the  simulation  used  fulfills  the  four  critical  elements  of  OT&E  and  clearly  represents 
a  realistic  operational  evaluation  of  the  weapon  system.  M&S  applications  must  be  accredited 
for  each  specific  application.  Because  of  the  VV&A  unique  requirement,  there  is  no  listing  or 
catalog  of  standard  M&S  applications  available  for  use  in  the  OT&E  process.  The  W&A 
conducted  for  one  operational  test  does  not  apply  to  another  operational  test.  Integration  of  M&S 
in  OT&E  requires  advanced  detailed  planning  that  allows  coordination  time  for  VV&A. 

(OPTEC  73-21-1,1993) 

1.  SUMMARY 

The  U.S.  and  the  Army  are  facing  a  dynamic  international  environment.  The  acquisition 
process  is  the  primary  means  through  which  the  Army  procures  new  weapon  systems  and 
technology  to  maintain  the  “technological  advantage”  in  this  dynamic  environment.  The  Army 
acquisition  process  is  designed  to  field  weapon  systems  that  will  provide  the  Army  with  a 
“technology  advantage”  in  a  dynamic  warfare  environment.  Within  the  acquisition  process, 
OT&E  assesses  the  effectiveness  and  suitability  of  systems,  concepts,  doctrines,  and  tactics  in 
realistic  combat  environments.  The  results  of  OT&E  assessments  have  significant  impact  on  the 
acquisition  of  major  weapon  systems  and  have  visibility  at  the  Congressional  level  and  the  top 
echelons  of  the  Army.  M&S  techniques  can  be  applied  in  OT&E  to  support  weapon  system 
acquisition.  Utilizing  M&S  to  assist  in  OT&E  can  aid  in  fighting  the  current  and  future  cost 
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constraints  that  negatively  impact  on  the  scope  of  “live”  testing.  Currently,  M&S  takes  many 
forms  that  can  span  the  acquisition  process  to  support  the  PM  in  controlling  cost,  schedule,  and 
performance. 
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m.  ARMY  MODELING  AND  SIMULATION 


A.  GENERAL 

The  highest  echelons  of  Army  leadership  recognize  the  need  to  use  advanced  Modeling 
and  Simulation  (M&S)  capabilities  for  weapon  systems  acquisition  in  the  post-cold  war  era.  In  the 
United  States  Army  Posture  Statement  FY95,  The  Honorable  Togo  D.  West,  Jr.,  Secretary  of  the 
Army  stated:  "The  Army  will  maintain  technological  superiority  through  pursuit  of  promising 
advanced  technologies  and  concepts,  developing  new  systems  when  existing  systems  have  reached 
the  end  of  their  useful  Uves  or  when  a  new  system  offers  a  revolutionary  combat  capability.  We 
will  exploit ...  simulation  to  support  acquisition  testing,  which  will  reduce  acquisition  costs." 
(West,  1994,  p.87)  Currently,  the  Army  is  using  over  500  models  and  simulations  in  the 
acquisition  process  to  support  OT&E.  (STRICOM  Homepage,  January  1997) 

This  chapter  describes  M&S  concepts  and  their  application  in  OT&E.  M&S  concepts  will 
be  explained  in  terms  of  classes,  hierarchies,  and  hybrid  applications.  These  chosen  M&S 
applications  are  not  all-inclusive,  but  are  representative  of  the  technology  available.  The  M&S 
concepts  and  applications  discussed  are  currently  available  or  are  being  planned  for  use  by  the 
U.S.  Army. 

B.  CLASSIFICATION  OF  MODELS  AND  SIMULATION 

In  1992,  the  Defense  Science  Board  approved  the  classification  of  models  and  simulations 
depicted  in  Figure  6.  This  classification  is  useful  in  applying  M&S  to  Army  OT&E  because  it 
allows  a  simple  break-down  of  complex  technologies.  M&S  applications  range  fi'om  simple 
models,  using  only  one  class  of  M&S,  to  complex  models  that  integrate  all  three  classes 
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1.  Constructive  Models  and  Simulations 


The  models  and  simulations  contained  within  the  constructive  class  currently  represent 
the  predominant  form  of  M&S  used  in  support  of  the  acquisition  process.  Within  the  acquisition 
process,  the  uses  of  constructive  M&S  include  design  and  engineering  trade-offs,  cost, 
supportabiliK,  operational  and  technical  requirements  definition,  and  operational  effectiveness 
assessments.  Constructive  M&S  consists  of  computer  models,  computer  war  games,  and 


con:!nu:erizea 


liCal  tools  which  are  used  b}’  decision  makers  to  evaluate  technical 


specifications  and  to  identify  w'eapon  system  shortfalls.  Constructive  models  are  used  to 


represent  a  system  and  its  employment  at  different  levels  of  detail.  At  the  lowest  levels, 
constructive  M&S  are  used  in  engineering  design  testing,  costing,  and  system  performance 
calculations.  At  the  highest  level,  constructive  M&S  provides  information  to  identify  mission 
short  falls  and  to  support  operational  effectiveness  testing.  (Piplani,  1994,  p.  4-2) 

Constructive  M&S  operates  either  with  or  without  human  interaction.  Without  human 
interaction,  they  may  be  run  in  multiple  iterations  to  provide  higher  statistical  confidence  in 
testing.  With  human  interaction,  they  validate  weapon  system  operational  tests  within 
specification  and  design  requirements. 

2.  Virtual  Simulation 

Virtual  simulation  brings  the  weapon  system  and  its  operator  together  in  a  synthetic  or 
simulated  environment.  Synonyms  are  often  used  in  describing  virtual  simulation,  which  include 
the  terms  human-in-the-loop,  man-in-the-loop,  warfighter-in-the-loop,  or  person-in-the-loop. 
Virtual  simulations  run  in  real  time,  enhancing  the  data  gained  from  each  test  iteration.  The 
comparatively  lower  cost  of  virtual  simulation  makes  it  an  attractive  testing  method  for  the 
program  manager.  As  an  example,  a  virtual  weapon  system  simulator  may  provide  a  virtual,  but 
nearly-real,  crew  compartment  with  the  necessary  equipment,  controls  and  display  panels.  A 
computer-generated  synthetic  environment  is  then  displayed  on  a  screen  in  front  of  the  crew  and 
on  the  crew  compartment  instrumentation.  Motion  of  the  platform  is  created  by  computer 
simulation  to  replicate  the  weapon  system  dynamics.  Sounds  of  the  system  and  equipment  can 
also  be  duplicated.  The  operators  are  thereby  immersed  in  an  environment  created  by  simulation 
that  to  the  user  (or  tester)  looks,  feels,  and  behaves  like  the  real  weapon  system.  When  using 
this  type  of  M&S  in  testing,  the  tester  operates  the  equipment,  receives  commands,  and  controls 
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the  weapons  just  as  in  actual  combat.  (Piplani,  1994,  p.  4-3) 

Virtual  simulations  provide  better  understanding  of  human  reactions,  decision  processes, 
and  man-machine  interfaces.  Linked  to  other  simulators,  the  interaction  of  multiple  weapon 
systems  can  be  examined,  leading  to  changes  in  tactics  or  refining  weapon  system  designs.  These 
simulations  also  provide  powerful  tools  for  testing  actual  system  hardware  and  software  within 
realistic  environments  for  early  OT&E.  Additionally,  virtual  simulation  can  provide  a  platform 
for  crew  training  prior  to  OT&E. 

a.  Virtual  Prototypes 

A  more  advanced  type  of  virtual  simulation  is  on  our  doorstep—  virtual 
prototyping.  In  this  realm,  a  three-dimensional  electronic  virtual  mockup  of  a  weapon  system 
allows  an  individual  to  interface  with  a  realistic  computer-simulated  synthetic  environment.  The 
representation  is  solely  a  computer  simulation  rather  than  actual  hardware  and  may  be  applied  in 
earh  OT&E  to  evaluate  concepts  and  human-machine  interfaces.  Additionally,  it  allows 
designers,  logistics  engineers,  and  manufactiinng  engineers  to  interface  with  the  weapon  system. 
Such  an  approach  supports  Integrated  Product  and  Process  Development  (IPPD)  and  concurrent 
engineering,  by  providing  a  common  platform  from  which  all  functional  disciplines  can  work. 

The  concept  of  the  designer,  operator,  maintainer  and  manufacturer  all  interacting  with  the  same 
realistic  three-dimensional  representation  of  the  weapon  system  will  become  more  prevalent  in 
future  acquisition.  (Piplani,  1994,  p.  4-4) 

3.  Live  Exercises 

Live  exercises  consist  of  testers  and  soldiers  using  equipment  under  environmental, 
operational,  and  force  structure  conditions  that  replicate  actual  combat.  Live  exercises  provide  a 


testing  ground  to  provide  “real”  data  on  actual  hardware  and  software  operating  in  a  near-combat 
environment.  The  introduction  of  multiple  platforms  allows  for  evaluation  of  interoperability. 
However,  assembling  the  personnel  and  equipment  to  conduct  a  live  exercise  is  a  resource 
intensive  enterprise  requiring  time,  funds,  and  people.  Data  obtained  during  live  exercises  can  be 
used  to  validate  earlier  models  and  simulations  used  in  testing.  This  form  of  testing  reinforces 
virtual  simulation,  ensuring  the  weapon  system  supports  the  mission  need.  Constructive  and 
virtual  simulations  conducted  prior  to  live  exercises  can  be  used  to  plan  the  tests,  identify  critical 
issues,  and  train  the  participants.  Constructive  and  virtual  simulations  may  also  be  used  to 
analyze  results  after  the  live  exercises  or  to  augment  tests  that  may  not  be  feasible  to  conduct  in 
live  exercises  due  to  safety  or  environmental  reasons.  (Piplani,  1994,  p.  4-5) 

Before  proceeding  further,  the  reader  is  cautioned  that  not  every  application  of  M&S  fits 
neatly  into  one  of  the  above  classes.  The  lines  among  the  various  classes  of  M&S  are  becoming 
blurred  as  technology  advances,  allowing  interoperability  among  the  classes  of  M&S.  Hybrid 
types  of  models  and  simulations  are  bridging  the  gaps  among  two  or  more  classes  in  support  of 
OT&E. 

C.  HIERARCHY  OF  MODELS  AND  SIMULATIONS 

M&S  supports  the  OT&E  process  across  a  wide  span  of  activities  ranging  from  design  to 
operational  effectiveness  assessments.  This  assortment  of  tasks  requires  a  suite  of  models  and 
simulations  with  differing  levels  of  detail,  focused  at  particular  applications.  The  hierarchy  of 
M&S  depicted  in  Figure  7  defines  the  differing  levels  of  detail  and  their  application  in  OT&E. 
Alongside  the  hierarchy,  a  force  level  and  system  work  breakdown  structure  (WBS)  are  shown  to 
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indicate  the  system  level  that  corresponds  with  the  level  of  testing  performed.  (Piplani,  1994,  p. 


Hierarchy  of  Models  and  Simulations 
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Figure  7.  Hierarchy  of  Models  and  Simulations  (Piplani,  1994,  p.  4-6) 


1.  Engineering  Level  Models  and  Simulations 

Engineering  level  models  and  simulations  provide  the  basis  for  testing  design  trade-offs 
at  the  component,  subsystem,  and  system  levels.  Engineering  level  M&S  support  development 
of  technical  design  specifications  and  OT&E  requirements.  At  the  engineering  level  there  are 
literally  thousands  of  models  and  simulations  including:  (Piplani,  1994,  p.  4-7) 

Basi.-;  n;odcls  testing  phenomenology  such  as  aerod3-namics,  fluid  Bow. 
hydrodynamics,  heat  transfer,  acoustics,  fatigue. 
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*  Physics-based  models  of  components;  subsystem;  and  systems  for  design, 

performance,  costing,  scheduling  and  supportability. 

Examples  of  engineering  level  M&S  are  cost,  support,  and  manufacturing  models.  Cost 
models  generate  development,  production,  operations,  and  support  costs.  Support  models 
generate  reliability,  availability,  and  maintainability  data.  Manufacturing  models  generate 
information  on  producibility  of  a  particular  design,  as  well  as  simulation  of  maintenance  work 
flow  and  requirements. 

Engineering  level  M&S  evaluate  performance  capabilities,  often  termed  Measures  of 
Performance  (MOP).  Examples  of  these  measures  for  a  missile  system  may  include  radar 
acquisition  range,  miss  distance,  range,  payload  or  speed.  Such  performance  parameters  might  be 
used  in  the  system,  development,  and  testing  specifications.  The  representations  of  a  weapon 
system  in  a  higher  level  M&S  have  their  basis  in  engineering  level  models.  It  is  in  higher  level 
models  and  simulations  that  the  actual  impacts  of  weapon  system  performance  on  combat 
effectiveness,  as  part  of  OT&E,  are  evaluated. 

2.  Engagement  Level  Models  and  Simulations 

Engagement  level  models  and  simulations  represent  the  system  in  a  combat  scenario, 
such  as  one-on-one  or  many-on-many.  This  level  of  simulation  evaluates  the  effectiveness  of  an 
individual  weapon  system  against  a  specific  target  or  enemy  threat  system.  These  models  provide 
survivability,  vulnerability,  and  lethality  results  as  Measures  of  Effectiveness  (MOEs)  for  the 
system. 
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The  outputs  from  the  use  of  engagement  level  M&S  indicate  the  effectiveness  of  some 
w^eapon  systems  in  an  engagement  scenario.  Examples  for  a  missile  system  may  include 
probability  of  kills  and/or  probability  of  hits.  The  uses  of  engagement  level  M&S  assist  the 
program  mangers  in  identifying  system  effectiveness  and  performance  requirements  from  the 
Mission  Need  Statement  (MNS)  and  Operational  Requirements  Document  (ORD).  Additionally, 
engineering  level  M&S  is  used  to  conduct  system  level  performance  tradeoffs  and  to  evaluate 
weapon  systems  tactics. 

D.  HYBRID  MODELS  AND  SIMULATIONS 

Up  to  this  point,  the  M&S  classes  (constructive,  virtual  and  live)  and  a  hierarchy  of 
models  and  simulations  have  been  illustrated.  In  many  applications,  combinations  among  two  or 
more  classes  of  M&S  are  used,  resulting  in  hybrid  models  and  simulations.  A  hybrid  application 
of  M&S  might  employ  constructive  models  to  represent  a  threat  in  conjunction  with  actual  (live) 
system  hardware  and  software.  Examples  of  such  hybrid  applications  include  physical  attributes 
simulation,  stimulators,  and  hardware/software  in-the-loop  (HW/SWIL)  simulations,  and 
advanced  distributed  simulations  (ADS). 

1.  Physical  Attributes  Simulation 

Phy  sical  simulations  refer  to  a  physical  representation  of  the  actual  operating 
environment  (e.g.,  temperature,  humidity,  shock,  vibration).  Examples  of  physical  simulation 
include: 

Munitions  shock  and  vibration  testing  in  a  simulated  environment. 

*  Survivability/  vulnerability  evaluations  using  prototype  structures  and  simulated 

environmental  conditions,  such  as  high  speed  airflow,  temperature,  and  altitude. 
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*  A  firing  impulse  simulator  (a  hydraulically-operated  ram)  used  to  provide  an  impulse  to 
the  gun  barrel  to  physically  rephcate  the  shock  of  a  round  being  fired  for  durability  and 
shock  testing  of  the  artillery  piece. 

*  A  test  facility  that  simulates  dynamic  loads  and  motion  for  evaluation  of  tracked  and 
wheeled  vehicle  suspension  systems. 

These  are  just  some  representative  samples  of  physical  simulations.  Physical  simulation 
can  be  used  throughout  the  OT&E  process  at  component,  subsystem  and  system  level  for 
evaluating  Critical  Operational  Issues  (COI).  The  data  collected  fi-om  these  simulations  is  useful 
for  validating  earlier  models  and  simulations.  (Piplani,  1994,  p.  4-10) 

2.  Stimulation 

Stimulations  are  “non-live”  inputs  given  to  weapon  systems  to  test  the  weapon  system, 
just  as  if  the  “real”  input  were  present.  Stimulations  are  used  because  in  many  instances  the 
actual  signals  representing  the  “live”  environment  are  not  available.  These  include,  for  example,  a 
radar’s  return  from  a  target,  a  signal  from  another  weapon  system,  or  background  noise  fi'om  the 
environment  in  which  the  weapon  system  must  operate.  Such  stimulations  may  come  fi’om  other 
computer  models,  virtual  simulations  or  fi-om  live  instrumented  tests. 

3.  Hardware/Software-In-The-Loop 

The  hardware/software-in-the-loop  (HW/SWIL)  simulations  typically  span  multiple 
classes  of  simulations.  Figure  8  depicts  HW/SWIL  integration.  The  HW/SWIL  includes  actual 
hardware  and  software  with  M&S  to  demonstrate  the  capability  of  a  weapon  system  or 
subsystem.  An  HW/SWIL  simulation  has  proven  to  be  an  important  tool  in  system  development, 
test  and  operational  support.  (Piplani,  1994,  p.  4-11) 


29 


In  support  of  test  programs,  the  HW/SWIL  simulations  allow  for  pretest  simulation  to 
identify  test  conditions.  As  a  risk  reduction  measure,  they  are  used  for  verification  of  actual 
hardware  and  software  interaction.  These  simulations  are  also  used  to  conduct  post  test  analysis 
and  to  fill  a  test  matrix  for  conditions  that  are  either  not  testable  or  for  which  no  test  assets  are 
available.  The  HW/SWIL  simulations  allow  early  identification  and  correction  of  developmental 
problems  and  allow  one  to  identify  and  focus  live  tests  toward  critical  issues.  With  the 
increasing  reliance  on  software  \vithin  weapon  systems,  and  regular  software 


Figure  8  Hardware/Software-ln-The-Loop  Simulation  (HW/SWIL)  (Piplani,  1994,  p4-l  1) 


changes  to  enhance  or  modify-  performance,  HW/SWTL  has  become  a  primary  tool  in  testing  the 
efiects  of  software  changes  or.  performance. 
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4. 


Advanced  Distributed  Simulation 


a.  Distributed  Interactive  Simulation 

One  of  the  challenges  in  integrating  constructive,  virtual,  and  live  simulation 
technologies  is  the  use  of  dissimilar  computer  systems  and  simulators  languages.  Many  models 
and  simulations  were  designed  independently  and  were  not  designed  to  communicate  with  one 
another.  The  integration  of  simulation  capabilities  using  a  common  set  of  standard  terms  and 
protocols  are  needed  to  allow  different  computer  systems  and  simulators  to  communicate.  The 
standardization  of  terms  is  achieved  by  Distributed  Interactive  Simulation  (DIS)  Protocol  Data 
Units  (PDUs).  (Crouch,  1994,  p.56) 

DIS  creates  a  sjoithetic  environment  within  which  humans  may  interact  through 
simulation  at  multiple,  networked  sites  using  compliant  architectures,  modeling,  protocols, 
standards,  and  databases.  DIS  is  the  next  generation  of  distributed  simulation  evolving  from  the 
Defense  Advanced  Research  Project  Agency  (DARPA)  research  project  of  the  1980’s  known  as 
Simulation  Network  (SIMNET).  (1ST,  1993,  p.4) 

DIS  will  take  advantage  of  current  and  future  simulations  manufactured  by 
different  organizations.  The  first  step  in  achieving  this  interoperability  is  to  develop  a  single 
communications  protocol.  There  must  be  a  standard  set  of  messages  that  communicate  the  states 
of  simulated  and  real  entities,  their  interactions,  and  among  host  computers.  This  information  is 
communicated  through  DIS  PDUs.  The  DIS  is  in  its  initial  stages  with  many  obstacles  yet  to 
overcome.  (1ST,  1993,  p.8) 

b.  Defense  Simulation  Internet 

DIS  operations  are  supported  by  a  communication  system  known  as  the  Defense 
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Simulation  Internet  (DSI).  This  communication  system  was  developed,  and  is  currently  operated, 
by  the  DARP A.  The  DSI  consists  of  commercial  telephone  circuits  over  existing  networks  with 
nodes  at  user  locations  and  strategically  placed  switching  nodes.  A  central  controlling  facility  is 
located  in  Chicago,  Illinois.  Connectivity  is  established  through  military  and  civilian 
corrununication  satellites  to  allow  worldwide,  simultaneous  DSI  operations. 

Each  location  connected  to  the  DSI  network  is  called  a  DSI  node.  There  are 
approximately  30  DSI  nodes  supporting  all  Services'  command  posts.  Battle  Simulation  Centers, 
test  beds.  Battle  Labs,  research  centers,  unified  commands,  and  civilian  contractors  that  support 
the  military-.  The  DSI  is  expected  to  expand  over  the  next  year  with  approximately  25  additional 
sites.  Currently',  the  Army  operates  six  core  locations  which  make  up  the  primary  backbone  of 
the  DSI  nehN-ork.  The  Army  has  two  TRADOC  Battle  Lab  nodes  on  the  DSI;  Fort  Knox  and  Fort 
Rucker,  with  Fort  Rucker  designated  as  a  Battle  Lab  support  facility.  Eventually  the  Army 
wishes  to  operate  six  to  eight  Army  Battle  Lab  nodes  with  additional  communications  nodes  at 
major  commands  throughout  the  Army.  (STRICOM  Homepage,  15  Januaiy  1997) 

Simulation  tools  and  methodology  integration  between  the  Battle  Labs  and 
industry  offer  a  new  area  of  innovation  that  has  potential  to  support  the  acquisition  process.  The 
use  of  concurrent  engineering  principles  reduces  development  time  and  speeds  the  acquisition 
process.  Virtual  prototy'pes  will  enhance  early  design  and  manufacturing  tradeoffs,  leading  to 
effecti\-e  OT&E.  Eventually  the  manufacturing  process,  the  actual  weapn  system,  and  the 
system’s  performance  may  all  be  modeled,  integrated,  and  tested  before  the  first  piece  of 
hardware  is  built.  (BLG,  1993,  p.6)  Synthetic  environments  will  not  completely  replace 
hardware  testing.  However,  given  the  increased  costs  of  hardware  development  and  testing, 


contrasted  with  the  decreased  costs  and  advancing  technology  simulators,  synthetic  environments 
become  ever  more  appealing.  (BLG,  1993,  p.7) 

E.  SUMMARY 

The  Army  is  moving  to  take  advantage  of  advanced  M&S  opportunities  and  the  potential 
cost  savings  M&S  offers  in  a  fiscally  constrained  post-cold  war  era.  As  a  vital  part  of  the 
acquisition  process,  OT&E  can  integrate  M&S  concepts  and  applications  to  address  recurring 
problems  and  shortcomings  in  cost,  schedule,  and  OT&E  performance.  The  classes  of  M&S  can 
be  included  in  OT&E  planning  to  help  the  program  manager  in  reaching  cost,  schedule,  and 
performance  goals.  With  the  high  cost  of  live  testing,  the  use  of  less  resource-intensive  forms  of 
testing,  such  as  M&S,  is  a  smart  preparation  tool.  As  an  integral  part  of  OT&E  planning  and 
support,  M&S  allows  a  program  manager  to  use  costly  and  valuable  assets  more  efficiently. 

The  Army  will  link  M&S  technology  components  together  through  networks  such  as  the 
Defense  Systems  Internet  (DSI)  to  create  environments  such  as  the  Distributed  Interactive 
Simulation  (DIS).  Relying  heavily  upon  simulation  and  modeling  technologies,  senior  Army 
leaders  will  use  M&S  as  a  tool  to  help  save  money  by  speeding  up  the  introduction  of  promising 
new  weapon  systems.  The  Army's  leadership  is  setting  policies  that  endorse  M&S  use  in  today’s 
costly  acquisition  process. 


rV.  ARMY  SIMULATION  -  POLICIES  AND  REGULATIONS 


A.  GENERAL 

The  Operational  Test  and  Evaluation  (OT&E)  process  is  guided  by  numerous  policies  and 
regulations.  The  provisions  of  these  mandates  and  guiding  vehicles  directly  affect  how  OT&E 
for  a  weapon  system  is  planned,  conducted,  and  analyzed.  The  role  of  M&S  in  the  OT&E 
process  is  directed  by  the  policies  and  regulations  guiding  OT&E.  Initial  efforts  in  providing 
effective  policy  and  regulatory  guidance  lay  the  foundation  for  refining  the  Army’s  future 
processes  for  managing  its  M&S  investment.  Policy  and  regulatory  guidance  define  how  OT&E 
can  encompass  M&S  to  streamline  the  OT&E  process.  This  chapter  describes  the  general  Army 
policy  and  regulations  that  affect  the  use  of  M&S  in  OT&E. 

B.  ARMY  AND  M&S  POLICIES 

The  Army  is  adopting  policies  to  take  advantage  of  advanced  M&S  opportunities.  Army 
leaders  are  setting  the  M&S  policy  cornerstones  as  evidenced  by  Army  Chief  of  Staff,  General 
Gordon  Sullivan’s  statement; 

You  need  to  know  that  we  will  use  simulation  techniques  throughout  the  Army’s 
acquisition  process.  We  will  determine  needs  in  large-scale,  simulation-supported 
exercises  that  allow  us  to  consider  alternative  solutions  that  meet  our  needs.  We 
will  use  drawings,  diagrams,  and  3-dimensional  models  generated  by  computers, 
put  them  in  constructive  or  virtual  environments,  and  test  alternatives  both 
technically  and  tactically.  The  most  promising  technologies  will  be  tested  by  real 
soldiers,  first  in  reconfigurable  crew  stations,  then  in  full  scale  simulations.  Final 
designs,  production  and  assembly  steps  will  also  be  simulated  in  virtual  factories 
before  actual  prototypes  are  made.  Then  the  actual  and  virtual  prototypes  will  be 
tested  simultaneously  to  discover  potential  problems  before  production  begins. 

Tactics,  techniques,  and  procedures  are  also  developed  along  with  the  system  so 
that  the  system  is  fully  ready  for  use  when  produced.  (Sullivan,  1993) 
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Lieutenant  General  William  H.  Forster,  the  Military  Deputy  to  the  Assistant  Secretary  of 
the  Army  (Research,  Development  and  Acquisition),  further  refined  the  M&S  policy  in  his  24 
May  1993  Memorandum  for  the  Deputy  Commanding  General,  Army  Materiel  Command  and 
all  Program  Executive  Officers.  Under  the  subject  of  "Simulation  Support  to  the  Army,”  LTG 
Forster  stated. 

The  Army  Science  Board  and  the  Defense  Science  Board  have  recently  studied 
the  potential  improvements  to  DoD  acquisition  offered  by  advanced  simulation, 
particularly  Distributed  Interactive  Simulation  (DIS).  Both  concluded  that 
simulation  can  improve  acquisition  from  concept  to  fielding  through  such 
innovations  as:  virtual  prototyping;  engineering  simulation;  linking  of 
constructive,  virtual  and/or  live  simulations;  assisting  the  user  in  execution  of 
experiments  in  employment  tactics;  user  test  design  and  critical  issue 
identification;  and  improved  training  prior  to  fielding.  (Forster,  1993,  p.l) 

Policies  integrating  M&S  into  the  acquisition  process  have  been  initiated  from  the  top 

levels  of  the  Army.  Additionally,  orgamzations  have  been  established  and  charged  with  pursuing 

new  technologies  and  investigating  concepts  using  an  integrated  M&S  approach.  Concepts  for 

integrating  M&S  into  the  OT&E  portion  of  the  acquisition  process  are  coordinated  through  three 

primary'  organizations:  The  Army  Model  and  Simulation  Management  Office  (AMSMO), 

Operational  Test  and  Evaluation  Command  (OPTEC),  and  Army  Material  Systems  Analysis 

Activity  (AAfSAA). 

The  Army  Model  and  Simulation  Management  Office  (AMSMO)  is  the  focal  point  for 
establishing  and  maintaining  the  methodology  by  which  to  identify  and  share  information  among 
agencies,  organizations,  and  activities  in  the  Army  community'  who  use  or  develop  M&S.  The 
AMSMO  promulgates  Army  M&S  policy,  publishes  guidance,  and  administers  the  Army  Model 
Improvement  Program  (AMIP)  and  Simulation  Technology  Program  (SIMTECH).  Additionally 
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AMSMO  develops  and  publishes  the  Anny  Modeling  and  Simulation  Master  Plan,  implementing 
procedures  such  as  the  verification,  validation  and  accreditation  (VV&A)  of  Army  models  and 
simulations.  (AMSMO  Homepage,  1996) 

The  Operational  Test  and  Evaluation  Command  (OPTEC)  is  the  Army’s  lead  agent  for 
operational  test  and  evaluation.  OPTEC  is  charged  with  integrating  M&S  throughout  the  testing 
process.  OPTEC’s  goal  is  to  apply  cost-effective  techniques  to  situations  of  limited  testability. 
Additionally,  OPTEC  incorporates  the  notion  of  Distributed  Interactive  Simulation  (DIS)  through 
the  use  of  constructive  and  virtual  simulations  in  operational  testing.  (OPTEC  Homepage,  1996) 

The  Army  Material  Systems  Analysis  Activity  (AMSSA)  maintains  configuration  control 
on  M&S  for  item  level  performance.  This  includes  one-on-one  system  performance,  many-on- 
many  system  combat  performance,  and  larger  war  gaming  simulations  for  OT&E.  AMSSA 
uses  M&S  to  provide  allied  and  threat  systems  performance  data  for  use  in  cost  and  operational 
effectiveness  testing  in  Army  studies  and  in  support  of  the  acquisition  process.  AMSSA  assists  in 
accreditation  and  provides  certified  systems  performance  data  characteristics  and  data  to  the 
Army  testing  community.  Additionally,  AMSSA  provides  W&A  support  to  OPTEC  and  the 
Army  Material  Command  (AMC).  (AMSSA  Homepage,  1996) 

C.  PRIMARY  REGULATIONS  GOVERNING  M&S 

Dramatic  advancement  in  modeling  and  simulation  have  made  its  use  in  the  OT&E 
process  more  attractive  to  many  program  managers.  Operational  testers  are  beginning  to  use 
M&S  to  support  OT&E.  Lawmakers  are  attempting  to  exploit  the  advances  in  M&S  by  adjusting 
or  creating  regulations  that  specify  the  use  of  M&S  in  OT&E.  Figure  9  depicts  the  primary 
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documents  that  govern  the  use  of  M&S  in  the  acquisition  process  and  OT&E.  Each  document 
provides  focus  to  guide  the  potential  M&S  user. 


Figure  9  Documents  and  Regulations 


1.  Title  10,  United  States  Code 

United  States  Code,  Title  10,  Chapter  138  has  two  sections  that  pertain  to  the  OT&E 
process.  Section  2399,  Operational  Test  and  Evaluation  of  Defense  Acquisition  Programs,  and 

section  2^66,  Major  Systems  and  Munitions  Programs:  survivability,  lethality,  and  operational 
testing. 


Section  2399  establishes  the  conditions  necessary  for  a  major  program  (acquisition 
c  ategory  I  and  II)  to  proceed  beyond  low  rale  initial  production  (LRIP).  This  provision  states  that 


a  major  defense  acquisition  program  may  not  proceed  beyond  LRIP  until  the  Initial  Operational 
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Test  and  Evaluation  (lOTE)  for  that  program  is  complete.  It  further  directs  that  the  Director  of 
Operational  Test  and  Evaluation  (DOT&E)  shall  analyze  the  results  of  major  systems'  OT&E. 
The  resulting  report  shall  state  the  DOT&E's  opinion  regarding  OT&E  adequacy  and  whether  the 
results  indicate  that  the  evaluated  system  is  effective  and  suitable  for  combat.  (USC,  1989, 
pp.230-231)  . 

Section  2366  directs  the  Secretary  of  Defense  to  ensure  that  no  major  system  proceeds 
beyond  TRIP  until  the  lOTE  survivability  and  lethality  testing  is  complete.  This  section 
specifically  defines  the  conditions  required  for  survivability  and  lethality  testing.  These 
conditions  include  the  requirement  for  live  fire  testing  on  representative  systems  and,  therefore, 
preclude  any  major  system  from  being  acquired  solely  on  the  basis  of  OT&E  conducted  using 
only  M&S.  (USC,  1989,  pp.581-582) 

2.  Department  of  Defense  Regulation  (DoD)  5000.2-R 

This  regulation  references  and  reinforces  the  provisions  of  Title  10,  Unites  States  Code. 

It  directs  that  OT&E  shall  be  designed  to  support  the  decision  to  proceed  beyond  TRIP  and 
establishes  four  objectives  for  OT&E  testing: 

1.  Provide  essential  information  for  assessment  of  acquisition  risk  and  for 
decision  making. 

2.  Verify  attainment  of  technical  performance  specifications  and  objectives. 

3.  Verify  that  systems  are  operationally  effective  and  suitable  for  intended  use. 

4.  Provide  essential  information  in  support  of  decision  making. 

With  regard  to  the  use  of  M&S  in  the  OT&E  process,  DoD  5000.2-R  interprets  Title  10  in 
a  strict  manner.  The  regulation  states  that  OT&E  does  not  include  an  operational  assessment 
based  exclusively  on  computer  modeling  and  simulation.  (DoD  5000.2-R,  1996,  p.  15) 
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3. 


.  DoD  M&S  Master  Plan 


The  DoD  M&S  Master  Plan  is  the  Department's  first  effort  in  directing,  organizing,  and 
concentrating  M&S  capabilities  and  efforts.  It  provides  a  means  to  achieve  a  common  technical 
and  policy  consensus.  Because  of  the  immense  breadth  and  scope  of  DoD  M&S  uses,  combined 
with  the  relative  immaturity  of  many  segments  of  the  larger  DoD  M&S  community,  a 
comprehensive  and  complete  master  plan  is  difficult  to  obtain.  This  plan  is  intended  to  be 
dynamic  and  flexible,  a  living  document  that  will  evolve  as  M&S  technology  matures  and 
consensus  develops  on  policy  and  programmatic  issues.  While  many  policy  and  technical  issues 
are  not  identified  or  resolved,  it  does,  how^ever,  provide  the  management  framework  and  policy 
direction  for  further  implementation  of  M&S  in  the  acquisition  process.  Over  time,  with  the 
active  participation  and  support  of  the  DoD  M&S  community,  the  DoD  M&S  Master  Plan  will 
mature  to  address  the  full  range  of  issues  confronting  M&S  applications  within  DoD. 

4.  Army  M&S  Master  Plan 

The  Army  M&S  Master  Plan  provides  information  on  the  modeling  and  simulation 
vision,  standards  development  process,  and  objectives  for  the  Department  of  the  Army.  It 
represents  a  statement  of  the  future  use  of  M&S  within  the  Army.  The  Army  M&S  Master  Plan 
adopts  the  same  M&S  objectives  as  the  DoD  M&S  Master  Plan.  Additionally,  the  Army  M&S 
Master  Plan  provides  guidance  to  program  managers  and  operational  testers  on  how  to  develop 
plans,  programs,  and  projects  to  incorporate  the  objectives  of  the  Army  in  relation  to  models  and 
simulations. 
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5.  Army  Regulation  5-11,  Army  Model  and  Simulation  Management 
Program 

Army  Regulation  5-11  (AR  5-11)  describes  what  types  of  M&S  fall  mder  the  purview  of 
this  military  regulation,  as  well  as  those  types  of  M&S  that  are  excluded.  Specifically,  M&S 
used  for  education  and  training,  analysis,  test  and  evaluation  (T&E),  research  and  development 
(R&D),  or  production  and  logistics  are  to  be  responsive  to  the  provisions  of  AR  5-11.  Also 
included  are  models  and  simulations  that  produce  input  for  other  models,  whose  results  are  then 
used  by  Army  decision  makers.  M&S  developed  and/or  used  by  contractors  or  federally  funded 
research  and  development  centers  (FFRDC)  in  support  of  Army  activities  must  also  comply  with 
the  provisions  of  AR  5-11.  Finally,  simulators,  semi-automated  forces  (SAFOR),  and  models  and 
simulations  that  operate  in  a  Distributed  Interactive  Simulation  (DIS)  environments  are  likewise 
included  in  this  grouping.  Models  and  simulations  not  falling  under  the  purview  of  AR  5-1 1  are 
system  training  devices,  those  embedded  in  weapons  systems,  machines  or  tools,  and 
instrumentation  that  does  not  communicate  outside  the  primary  weapon  system  or  device, 
a.  DA  PAM  5-11 

This  new  Department  of  the  Army  Pamphlet  (DA  PAM)  sets  procedures  for 
policy  contained  in  AR  5-1 1,  the  Army  Model  and  Simulation  Management  Program.  The 
objective  of  this  pamphlet  is  to  assist  the  M&S  developer,  proponent,  and  application  sponsor  in 
conforming  to  the  verification,  validation,  and  accreditation  (VV&A)  provisions  prescribed  in 
AR  5-11.  This  pamphlet  also  provides  guidance  for  the  development,  execution,  and  reporting  of 
all  VV&A  activities.  This  pamphlet  does  not  cover  data  certification  procedures,  but  it  does 
address  data  certification  in  reference  to  proper  M&S  use.  DA  PAM  5-1 1  outlines  all  of  the 
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documentation,  configuration  management,  certification,  and  verification  and  validation 
requirements  of  data  in  M&S. 

6.  Army  Regulation  73-1,  Test  and  Evaluation  Policy 

Army  Regulation  73-1  (AR  73-1)  directs  the  implementation  of  DoD  Directive  5000.1 
and  DoD  Regulation  5000.2-R.  This  regulation  describes  the  type  of  OT&E  application  to  be 
provided  in  support  of  each  phase  and  milestone.  It  links  T&E  to  the  acquisition  process  and 
directs  that  T&E  be  designed  to  support  the  acquisition  phases  and  milestones.  Within  this 
regulation,  OPTEC  is  charged  with  the  overall  management  of  Army  OT&E,  including  the 
integration  of  M&S  where  possible  to  reduce  cost,  save  resources,  and  verify  available  data, 
a.  DA  Pamphlet  73-1 

Department  of  the  Army  Pamphlet  73-1  (DA  Pam  73-1)  suggests  the  uses  of 
models  and  simulations  in  test  and  evaluation  (T&E).  M&S  will  be  considered  to  support  the 
de\'elopmental  and  operational  T&E  of  software  and  weapon  systems  as  they  proceed  through  the 
acquisition  life  cycle.  The  use  of  M&S  will  include,  but  not  be  limited  to,  the  identification  of 
test  parameters  and  drivers  for  field  tests;  determination  of  high  risk  areas;  prediction  of  test 
results;  assisting  in  the  allocation  of  scarce  test  resources;  and  the  assessment  of  system 
capabilities  in  situations  that  cannot  be  tested  due  to  safety,  cost,  or  other  constraints. 

Simulators,  emulators,  drivers,  and  stimulators  which  are  used  to  fully  exercise  weapon  systems 
under  operational  test  conditions  are  also  included.  DA  PAM  73-1  stipulates  that  M&S  used  for 
T&E  must  be  accredited  and  validated  prior  to  its  use  for  extrapolation  or  predicting  system 
performance  (including  software,  hardware,  man-in-loop).  The  degree  to  which  M&S  will 
augment  OT&E  to  assist  in  weapon  system  evaluations  and  assessments  will  be  documented  in 
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the  Test  and  Evaluation  Master  Plan  (TEMP). 

The  TEMP  is  the  principal  document  for  test  scheduling  and  planning  for  the 
program  manager  and  the  T&E  community.  TEMP  planning  aids  decision  makers  by  verifying 
that  the  weapons  systems  have  attained  their  technical  performance  specifications  and  objectives. 
Specifically,  the  TEMP  ensures  that  testing  will  meet  operational  effectiveness  and  operational 
suitability  requirements  as  designed  by  the  user.  Additionally,  the  TEMP  addresses  whether 
existing  M&S  will  be  used  or  whether  new  M&S  will  be  developed  to  support  OT&E. 

D.  REGULATION  OVERVIEW 

Army  M&S  regulations  provide  the  flexibility  to  allow  each  acquisition  program  to  tailor 
a  T&E  strategy  applying  M&S  to  achieve  maximum  program  performance.  Operational  test 
strategies  must  be  generated  concurrently  with  the  acquisition  strategy  to  assure  that  M&S  is  an 
integral  part  of  the  acquisition  program.  Modeling  and  simulation  will  be  considered  to  support 
the  technical  and  operational  T&E  of  all  systems  (hardware  and  software)  as  they  proceed 
through  the  acquisition  life  cycle.  Models  and  simulations  used  for  OT&E  must  be  accredited 
and  validated  prior  to  their  use  for  extrapolation  or  predicting  system  performance.  The  use  of 
M&S,  whether  existing  or  new,  will  be  documented  in  the  Test  and  Evaluation  Master  Plan 
(TEMP) 

E.  SUMMARY 

The  continuing  revolution  in  advanced  technologies  has  widened  the  opportunity  for 
using  M&S  in  OT&E.  This  technical  revolution  has  increased  the  number  of  users  of  Army 
M&S  to  the  point  that  M&S  touches  almost  all  aspects  of  the  acquisition  process.  To  support 
the  growing  use  of  advanced  technology,  policy  and  regulations  have  been  established  to  provide 
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a  common  reference  and  standard  to  enhance  and  focus  concurrent  use  of  M&S  applications. 
Current  policy  and  regulations  are  designed  to  assist  the  program  manager  in  using  M&S 
applications.  The  development  of  clear  policies  and  regulations  have  led  to  a  better 
understanding  of  the  Army  M&S  management  and  an  increased  visibility  of  M&S  activities.  As 
advancements  in  M&S  technologies  continue,  the  Army  will  refine  current  regulations  and 
spawn  new  orgamzations  that  will  influence  policy.  These  new  organizations  will  update  polices 
and  drive  lawmakers  to  revise  current  regulations,  encouraging  the  use  of  M&S  in  OT&E. 

Investments  in  M&S  provide  a  critical  underpinning  for  a  variety  of  Army  missions,  from 
designing  a  future  force,  building  systems  and  doctrine  for  that  force,  and  finally  to  testing 
weapon  systems  for  that  force.  The  current  downsizing  of  the  Army  creates  pressure  to  reduce 
costs  at  every  level,  including  within  the  OT&E  process.  As  the  Army  continues  to  increase  the 
use  of  M&S  in  OT&E,  it  will  see  better  resource  management,  cost  savings,  and  enhanced 
verification  of  testing  data.  The  next  chapter  illustrates,  through  case  analysis,  the  results  of  the 
application  of  M&S  in  OT&E. 
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V.  CASE  ANALYSIS 


A.  GENERAL 

The  OT&E  community  has  a  most  difficult  role  in  defining  when,  where  and 
how  it  can  participate  outside  the  live  environment.  Operational  testers  find 
themselves  caught  between  United  States  Code,  Title  10  mandates  and  the  spiral 
of  diminishing  defense  dollars.  This  is  a  most  dangerous  position.  (Crouch,  1994, 

p.86) 

The  "most  dangerous  position"  refers  to  a  situation  where  the  OT&E  independent 
evaluator  (OPTEC)  does  not  integrate  M&S  into  the  testing  process  but  continues  to  use 
expensive  live  testing,  while  program  managers  (PMs),  materiel,  and  combat  developers  push  to 
incorporate  cost  effective  and  realistic  M&S  applications  for  OT&E  certification.  Current 
technological  advancements  in  M&S  provide  the  required  simulation  fidelity,  making  M&S  a 
viable  alternative  for  some  early  and  follow-on  portions  of  OT&E.  (Crouch,  1994,  p.87) 

This  chapter  examines  the  application  of  modeling  and  simulation  in  OT&E. 

Specifically,  three  weapon  systems  will  be  used  to  illustrate  the  viable  application  of  modeling 
and  simulation  in  OT&E.  The  three  primary  weapon  systems  are  the  Ml  A2  Abrams  Main  Battle 
Tank,  the  Javelin  Antitank  Missile  system,  and  the  AH-64D  Longbow  Apache  Helicopter. 

B.  METHODOLOGY 

This  analysis  will  focus  on  four  major  OT&E  issues:  test  design  validation,  resource 
constraints,  safety  and  environmental  concerns,  and  data  reliability  and  validity.  Each  of  the  four 
areas  is  critical  to  successful  OT&E.  These  major  OT&E  areas  of  concern  were  established  by 
conducting  interviews  with  experienced  OT&E  professionals  assigned  to  the  TEXCOM, 
STRICOM,  and  Program  Management  Offices.  Additional  interviews  were  also  conducted  with 


45 


personnel  from  OPTEC  and  TRAC  Monterey.  Interviews  were  critical  in  identifying  the  level  of 
M&S  integration  for  the  OT&E  programs  of  each  weapon  systems  examined.  The  areas  of 
concern  were  selected  when  at  least  two  of  the  three  OT&E  programs  experienced  similar 
problems  and  issues. 

1.  Overview  of  Issues 

The  four  OT&E  areas  of  concern  are  test  design  validation,  resource  constraints,  safety 
and  environmental  concerns,  and  data  reliability  and  validity.  The  general  concerns  of  each  area 
and  their  application  in  this  thesis  are  described  in  the  following  paragraphs. 

a.  OT&E  Test  Design  Validation 

Test  design  issues  were  categorized  as  OT&E  test  design  validation  concerns  when 
shortcomings  were  associated  with  the  baseline  test  design.  The  validation  process  ensures  that 
baseline  scenarios  and  test  execution  plans  satisfy  the  user  and  evaluator  requirements  for  data 
collection,  while  simultaneously  meeting  the  mandates  and  guidance  for  the  conduct  of  OT&E. 

A  desired  method  for  test  design  validation  is  the  use  of  a  pilot  test.  A  pilot  test  is 
a  rehearsal  of  test  events  using  test  participants,  fully  instrumented  candidate  systems,  data 
collection  resources,  test  ranges,  and  maneuver  areas.  A  pilot  test  is  useful  in  determining  if  the 
data  collected  address  the  criterion  under  consideration  for  specific  OT&E  events.  The  pilot  test 
advantages  include  validation  of  test  scenarios,  data  collection  plans,  test  facilities,  and  pretest 
training  with  no  test  report  impact.  Disadvantages  include  time  and  funding  requirements,  early 
user  familiarization  with  the  candidate  system,  test  facilities,  and  test  trial  scenarios. 


46 


b.  Resource  Constraints 


Shortcomings  associated  with  time,  personnel,  equipment,  and  funding  availability 
were  categorized  as  resource  constraints.  These  shortcomings  impact  the  OT&E  test  design  and 
scope  in  terms  of  realism,  environmental  diversity,  scale  of  threat  and  friendly  force  stractures, 
and  number  of  OT&E  event  iterations.  The  degree  to  which  resources  are  constrained  determines 
the  operational  scope  replicated  in  the  live  test,  including  both  the  type  and  amount  of  data 
available  for  evaluation. 

c.  Safety  and  Environmental  Concerns 

OT&E  events  that  were  eliminated  or  unrealistically  controlled  because  of 
potential  hazards  to  personnel  or  released  pollutants  into  the  environment  were  categorized  as 
safety  and  environmental  concerns.  As  the  range,  lethality,  and  destructive  ability  of  associated 
systems  all  increase  with  new  technologies,  safety  concerns  often  limit  the  use  of  the  actual 
weapon  system  to  very  controlled  circumstances  that  are  not  representative  of  a  combat 
environment.  Weapon  systems  that  create  radioactivity,  release  ozone  depleting  compounds,  and 
other  environmentally  hazardous  by-products  are  limited  to  controlled  scenarios. 

d.  Data  Reliability  and  Validity 

Issues  associated  with  test  data  interpretation  or  unusual  test  participant 
qualifications  were  categorized  as  data  reliability  and  validity  concerns.  Live  test  events  contain 
an  infinite  number  of  variables,  providing  an  opportunity  for  assigning  cause  to  undesirable 
outcomes,  eliminating  outliers,  and/or  biasing  data.  Obtaining  reliable  and  valid  data  for 
evaluation  is  difficult  in  an  uncontrolled  environment.  Data  reliability  and  validity  are  affected 
by  both  the  test  design  and  the  operational  experience  level  of  the  tester.  The  degree  to  which 
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test  participants  represent  typical  operational  users  impacts  projected  force  structure  and  tactical 
unitization  of  the  weapon  system  for  combat  operations. 

2.  Overview  of  Primary  Weapon  Systems 

a.  M1A2  Abrams  Main  Battle  Tank 

The  Ml  A2  is  the  second  major  block  improvement  to  the  Abrams  Main  Battle 
tank.  Enhancements  to  the  Ml  A1  Abrams  include  the  Inter  Vehicular  Information  System 
(rVIS),  Commander’s  Independent  Thermal  Viewer  (CITV),  Position/Navigation  System,  an 
Improved  Commander’s  Weapon  Station,  and  integration  of  the  vehicle  electronics  system 
through  the  use  of  dual  redundant  data  and  utility  bus  architecture.  This  weapon  system  is 
designed  to  be  fielded  in  armored,  mechanized  infantry,  and  cavalry  units,  ranging  from  platoon 
to  division  sized  elements.  (Patten,  January  1997) 

b.  Javelin  Anti-Tank  Missile  System 

The  Javelin  (formerly  known  as  the  Advanced  Antitank  Weapon  System- 
Medium)  is  a  man-portable,  fire  and  forget,  advanced  antitank  missile  system  capable  of 
defeating  all  known  armor  threats.  The  Javelin  is  replacing  the  shoulder-fired  Dragon  missile 
system  and  provides  extended  range  and  improved  attack  attitudes.  It  will  replace  most  Dragon 
systems  in  combat  and  combat  support  units  from  platoons  through  corps  level.  (Knox,  January 
1997) 

c.  AH-64D  Longbow  Apache  Helicopter 

The  AH-64  is  the  Army’s  premier  attack  aircraft,  providing  direct  and  indirect  fire 
support  to  both  ground  and  air  targets.  The  Longbow  Apache  is  an  upgrade  to  the  AH-64D 
Apache.  The  enhancements  include  improved  target  acquisition  and  firepower  effectiveness. 
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increased  survivability  and  lethality,  improved  adverse  weather  fighting  capabilities,  improved 
armament  (including  Longbow  Hellfire  missiles),  and  automated  systems,  engines,  and  fire 
control  radar.  The  Longbow  Apache  will  replace  attack  helicopters  currently  fielded  in  attack 
helicopter  units  at  battalion  through  corps  echelons.  (Wolfe,  January  1997  ) 

C.  ANALYSIS  OF  M&S  APPLICATION  IN  OT&E 

OT&E  problems  and  issues  within  each  of  the  identified  categories  are  detailed  and  the 
capabilities  of  M&S  applications  to  resolve  them  are  analyzed  in  the  following  paragraphs.  The 
analysis  is  organized  by  issue  category. 

1.  Test  Design  Validation 

The  operational  test  design  establishes  the  amount  of  data  and  the  circumstances  in  which 
data  are  collected.  Test  design  lays  the  foundation  for  the  amount  of  operational  realism 
achieved.  Creating  realistic  combat  conditions  in  OT&E  tests  can  be  hampered  by  many  factors: 
accurate  interpretation  of  Test  and  Evaluation  Master  Plan  (TEMP)  critical  issues,  poor 
understanding  of  the  environments  and  scenarios,  and  force  stmctures  required  to  replicate  a 
representative  combat  environment.  The  Ml  A2,  Javelin,  and  Longbow  Apache  test  designs  had 
shortfalls  in  test  design  validation. 

a.  M1A2  Main  Battle  Tank 

The  M1A2  live  test  used  an  operationally  smaller  force  structure  than  doctrinally 
employed  in  combat  operations.  Operational  testing  of  the  Ml  A2  tank  at  the  brigade  or 
divisional  levels  was  not  conducted.  Operational  testing  was  limited  to  one  battalion-sized  task 
force.  Battalions  deploy,  fight,  and  are  supported  differently  than  brigades  or  divisions.  The 
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reduced  scale  significantly  increases  data  assessment  extrapolation  risks  for  normal  combat 
operations. 

A  constructive  simulation,  such  as  Janus,  appears  to  have  application  in  addressing 
operational  test  design  shortfalls  similar  to  those  experienced  by  the  M1A2.  Constructive 
simulations  are  useful  for  assessing  operational  environments  and  combat  scenarios  beyond  the 
scope  of  limited  live  operational  testing.  Constructive  simulations  can  replicate  a  wide  range  of 
force  structure,  environments,  threat  forces,  and  combat  intensities.  In  using  constructive 
simulation  to  address  operational  test  design  issues,  live  testing  can  be  enhanced  across  the 
operational  spectrum  of  the  battlefield. 

TRAC  White  Sands  Missile  Range  (WSMR)  used  a  system  similar  to  Janus  to 
assist  in  the  M1A2  OT&E  test  design  validation.  Experiments  were  designed  on  a  Janus  based 
constructive  simulation,  and  experiments  were  iteratively  conducted  for  predictive  analysis. 
Actual  test  articles  were  instrumented  and  data  was  collected  on  live  test  trials.  Live  test 
iterations  were  replicated  on  the  simulations  and  then  correlated.  The  correlated  simulations  were 
then  used  in  other  environments  and  scenarios  within  the  M1A2  operational  spectrum  and 
analyzed.  The  insights  gained  from  these  simulations  were  used  to  assist  in  the  M1A2  live  test 
conducted  at  TEXCOM  Experimentation  Center  (TEC).  Live  OT&E  test  events  were  selected 
based  on  the  simulated  event  outcomes  that  indicated  a  high  probability  for  obtaining  the 
required  data  to  address  critical  issues.  (Payan,  1994) 

b.  Javelin  Missile 

Battlefield  operating  systems  normally  interoperating  with  the  Javelin  missile 
systems  were  not  included  in  the  Javelin  test  plan.  Javelin  is  deployed  in  conjunction  with  other 
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anti-armor  systems  including  tanks,  artilleiy,  and  close  air  support.  An  interoperational 
environment  was  too  expensive  and  impractical  to  replicate  in  a  live  operational  test,  and  was 
omitted  from  the  test  design. 

M&S  can  provide  a  method  for  addressing  interoperability  issues  such  as  those 
experienced  by  the  Javelin  during  OT&E.  Constructive  simulations  can  represent  other 
battlefield  operating  systems,  integrating  a  candidate  system’s  capabilities  so  that  interoperability 
issues  can  be  assessed.  Additionally,  virtual  simulations  can  provide  a  method  for  assessing 
interoperability  issues  with  personnel  operating  associated  battlefield  systems  within  doctrinal 
guidelines. 

Some  M&S  was  planned  into  the  Javelin  OT&E  test  design.  A  Model-Test-Model 
(MTM)  process  conducted  on  the  Javelin  program  successfully  correlated  live  OT&E  test  data 
with  models  generated  on  Janus.  Three  scenarios  with  six  missions  each  were  conducted  on  the 
Javelin  model,  including  offensive  and  defensive  Javelin  employments.  The  resulting  data 
provided  Javelin  employment  assessments  in  an  operational  environment.  OT&E  test  scenarios 
were  changed  to  incorporate  the  MTM  recommendations.  Integrating  MTM  efforts  into  future 
OT&E  test  designs  can  improve  test  scenarios  and  environments,  and  therefore,  enhance 
operational  assessments.  (IR  MTH,  1993) 

c.  AH-64D  Longbow  Apache 

During  comparative  performance  assessment  between  the  AH-64D  and  the  AH- 
64A,  threat  forces  were  realistically  deployed  and  represented  a  mid-intensity  conflict. 

(Stanfield,  March  1997)  However,  the  AH-64D  Longbow  Apache  OT&E  test  design  had  threat 
realism  shortfalls  for  high  intensity  conflicts.  After  the  lOT&E  comparative  assessement,  further 
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test  designs  did  not  replicate  many  of  the  threat  weapons  expected  in  a  combat  environment. 
Planned  threat  air  defenses  were  under-represented  and  lacked  realistic  electronic  jamming 
capabilities.  Additionally,  threat  air  assets  were  under-represented  and  excluded  from  many 
planned  test  scenarios. 

M&S  can  provide  methods  for  addressing  threat  realism  shortfalls  in  live  OT&E 
test  designs.  Constructive  simulations  can  replicate  the  full  range  of  threat  capabilities  expected 
in  a  combat  environment.  Additionally,  virtual  simulations,  incorporating  Semi-automated 
Forces  (SAPOR),  can  provide  a  means  to  include  man-in-the-loop  threat  forces  employed  by 
soldiers  using  doctrinally  correct  threat  tactics. 

M&S  applications  were  used  in  the  Longbow  Apache  OT&E  test  design  process. 
TEXCOM  validated  the  scenarios  for  the  test  plan  using  Janus  constructive  simulations.  In 
addition,  the  Longbow  Hellfire  missile  was  examined  in  various  electronic  warfare  counter¬ 
measure  situations  using  a  hardware-in-the-loop  simulation.  These  pretest  simulations  were  used 
to  establish  Hellfire  probability  of  hit  (Ph)  and  probability  of  kill  (Pk)  for  casualty  assessments. 
The  success  of  M&S  integration  in  the  Longbow  Apache  operational  test  indicates  that  M&S 
application  can  improve  the  test  design  process. 

d.  Test  Design  Summary 

In  many  operational  tests,  the  live  test  represents  only  a  small  part  of  the  actual 
combat  environment  for  the  weapon  system.  As  a  result,  the  data  collected  and  the  test 
evaluations  represent  only  a  part  of  the  operational  spectrum.  Data  for  evaluation  in  larger 
operational  assessments,  beyond  the  live  test  design,  must  be  extrapolated  to  address  the 
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weapon  system’s  combat  effectiveness  and  suitability.  As  the  scope  of  live  testing  is  reduced, 
increases  in  data  extrapolation  reduce  evaluation  validity. 

M&S  integration  into  the  OT&E  test  design  appears  to  provide  an  opportunity  for 
addressing  areas  that  are  beyond  the  scope  of  the  planned  live  test.  M&S  applications,  including 
MTM,  constructive  simulations,  and  virtual  simulations  can  enhance  the  OT&E  test  design 
process.  Candidate  systems  in  a  constructive  or  integrated  simulation  can  be  portrayed  at  every 
echelon  in  the  Army  structure,  under  varying  physical  and  threat  environments,  and 
interoperating  with  other  battlefield  and  logistics  systems. 

2.  Resource  Constraints 

A  declining  DoD  budget  adds  pressure  on  PMs  and  testers  to  limit  the  scope  of 
operational  testing.  Under  funding  and  limited  resources  contribute  to  reduced  realism  in  OT&E. 
Resource  constraints  had  a  serious  negative  impact  on  the  M1A2  and  Javelin  tests. 

a.  M1A2  Main  Battle  Tank  and  Javelin  Missile 

Due  to  lack  of  congressional  funding  and  limited  test  ranges,  the  Ml  A2  and  the 
Javelin  were  tested  at  the  small  unit  level.  The  horizontal  interoperability  and  vertical  impact  on 
higher  echelon  units  could  not  be  ascertained  by  the  OT&E  tests  conducted.  Resource  constraints 
limited  the  inclusion  of  associated  battlefield  operating  systems  and  support  elements  that  would 
normally  be  deployed  in  a  combat  environment.  Interoperability  evaluation  could  not  be 
addressed  from  test  data  collected.  Realism  in  live  testing  goes  beyond  the  small  unit.  For 
example,  recovering  and  evacuating  an  M1A2  tank  from  the  battlefield  to  the  corps  level  general 
support  maintenance  unit  requires  significant  recovery  and  transportation  assets  as  well  as  multi¬ 
echelon  coordination.  M1A2  OT&E  test  resource  constraints  did  not  allow  this  scenario  to  be 
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replicated  beyond  the  participating  battalion’s  organic  recovery  capabilities.  The  Javelin 
experienced  similar  problems  in  resupply  of  missiles  to  forward  troops  during  high  threat  combat 
environments.  In  the  Javelin  Live  test,  the  General  Support  unit  was  not  utilized. 

Interoperability  with  other  systems,  beyond  the  scope  of  live  testing,  can  be 
partially  assessed  through  M&S.  For  example,  the  TRAC  White  Sands  Missile  Range  (WSMR) 
simulations  addressed  M1A2  capabilities  in  various  environments  and  scenarios  beyond  the 
scope  of  live  testing.  Using  simulation  applications,  the  Ml  A2  could  be  portrayed  interoperating 
with  close  air  support,  artillery  and  any  other  battlefield  operating  system  including  critical 
logistics  support  activities.  The  capability  to  fully  replicate  interoperability,  both  horizontal  and 
vertical,  is  currently  outside  the  resource-constrained  OT&E  environment. 

Resource  constraints  have  a  considerable  impact  on  the  planning  of  live  tests. 
However,  M&S  has  application  in  addressing  the  OT&E  test  shortfall  created  by  resource 
constraints.  The  marginal  cost  of  M&S  generated  operational  events  is  extremely  low  compared 
with  those  same  events  conducted  in  the  live  environment.  An  example  of  the  wide  disparity 
betv'een  live  and  M&S  resource  requirements  involves  Ml  A1  main  battle  tank  prototypes.  In 
1984,  evaluations  of  possible  improvements  on  the  Ml  A1  Abrams  tank  were  carried  out  by  using 
real  tanks  in  a  live  environment.  The  effort  took  24  months  and  cost  $40  million.  A  later  effort  in 
1986  used  a  modified  aircraft  dome  simulator,  took  only  six  months  and  cost  $1  million.  In  1992, 
using  Distributred  Interactive  Simulation  (DIS),  four  variations  of  the  MlAl  Abrams  were 
operated  against  potential  threats,  taking  only  three  months  and  costing  $640, 000.  (Berry,  1992) 
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b.  Resource  Constraint  Summary 

Resource  limitations  impact  realism  and  reduce  the  scope  of  the  operational 
testing.  Reduced  realism  in  operational  testing  results  in  less  quantitative  and  more  qualitative 
evaluations  of  the  candidate  weapon  system.  Reduced  scope  in  operational  testing  increases 
data  extrapolation  for  larger  force-on-force  evaluations. 

Virtual  and  constructive  simulations  can  enhance  OT&E  tests  that  have  been 
negatively  affected  by  resource  constraints.  Virtual  simulation  can  provide  methods  for 
representing  threat  and  friendly  force  capabilities  that  are  too  expensive  or  impractical  to 
replicate  in  a  live  operational  test  environment.  Constructive  simulations  can  provide  a  means  to 
assess  portions  of  the  operational  spectrum  outside  the  scope  of  live  testing. 

3.  Safety  and  Environmental  Concerns 

Safety  and  environmental  impacts  of  new  technology  and  hazardous  complex  compounds 
adversely  affects  the  successful  creation  of  a  realistic  combat  environment  for  operational  testing. 
Peacetime  requirements  often  limit  the  amount  and  type  of  testing  that  can  be  accomplished, 
unlike  the  combat  environment  where  safety  and  environmental  side  effects  are  not  so  important. 
Safety  and  environmental  issues  affected  the  M1A2,  Javelin,  and  Lx)ngbow  Apache  OT&E  tests. 

a.  Range  Limitations 

A  common  problem  in  conducting  live  operational  tests  are  the  peqacetime  range 
limitations.  Range  limitations  affected  the  operational  employment  of  the  M1A2,  Javelin,  and 
Longbow  Apache  systems  during  OT&E.  The  ranges  used  had  well  marked  firing  limits  and 
oriented  the  crews  toward  target  positions.  Lasers,  for  all  but  the  Javelin  Missile,  were  restricted 
to  controlled  maneuver  and  air  spaces.  Controlled  maneuver  and  air  spaces  did  not  provide 
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representative  maneuver  space  or  engagement  ranges.  Environmental  concerns  eliminated 
operational  firing  of  depleted  uranium  M1A2  main  gun  rounds  due  to  radiation  hazards. 

Both  constructive  and  virtual  simulation  applications  provide  environments  that 
are  not  affected  by  these  types  of  safety  and  environmental  range  limitations.  As  a  result,  tests 
generated  through  M&S  can  avoid  such  biased  factors. 

b.  Crew  Safety  Requirements 

Both  PM  and  TEC  test  officers  strictly  followed  peacetime  crew  rest  requirements 
for  helicopter  pilots,  tank  crews,  and  ground  soldiers.  These  crew  rest  requirements  are  not 
representative  of  typical  combat  crew  duty  cycles,  which  are  not  constrained  to  peacetime 
training  restrictions.  Additionally,  during  OT&E  the  weapons  systems  were  not  tested  to  the 
normal  operational  readiness  factors  associated  with  combat  operations. 

In  a  protected  simulation  environment,  where  there  are  no  actual  threats  to  the 
safety  of  soldiers  and  where  safety  considerations  are  nearly  nonexistent,  M&S  applications  can 
be  used  to  assess  the  impact  of  extended  combat  operations  on  aircraft  and  ground  crews.  For 
example,  relatively  close  air-to-air  combat  involving  more  than  the  safety  constrained  two-on- 
two  force  ratios  are  not  probable  in  an  operational  environment.  Virtual  or  constructive 
simulations  provide  a  means  to  address  issues  involving  more  than  two-on-two  force  ratios. 

Crew  safety  concerns  make  simulation  a  viable  and  practical  method  for  obtaining  this  type  of 
operational  data. 

c.  Safety  and  Environmental  Concerns  Summary 

Safety  and  environmental  concerns  negatively  impact  on  the  ability  to  assess  the 
operational  effectiveness  and  suitability  of  candidate  weapon  systems.  Integrating  M&S  into  the 
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OT&E  process  provides  a  means  for  assessing  issues  in  an  environment  that  is  not  constrained  by 
safety  and  environmental  concerns.  Virtual  and  constructive  simulation  applications  can  be 
integrated  into  operational  tests  that  cannot  be  safely  or  environmentally  conducted  in  a  live 
operational  environment.  Additionally,  when  test  events  are  limited  in  number  due  to  safety  or 
environmental  concerns,  model-test-model  (MTM)  can  provide  a  method  to  extend  data  and 
enhance  operational  evaluations. 

4.  Test  Data  Reliability  and  Validity 

The  myriad  of  variables  present  in  a  live  test  environment  impact  on  data  reliability  and 
validity.  One  of  the  most  common  concerns  in  obtaining  reliable  and  valid  test  data  is  how  well 
test  personnel  represent  typical  users  at  the  operational  level.  The  Ml  A2  and  Longbow  Apache 
experienced  data  reliability  and  validity  concerns  due  to  crew  experience  levels  that  were  above 
typical  tactical  unit  proficiency  levels.  Experience  level  of  test  crews  centers  on  two  issues: 
l)Pre-training  of  test  personnel  for  the  specific  OT&E  event,  2)  operational  expereince  of  test 
participants.  OT&E  personnel  at  TEC  indicated  that  the  Ml  A2  and  Longbow  Apache  OT&E 
had  problems  associated  with  atypical  test  participants.  For  both  weapon  systens,  OT&E  test 
participants  were  tasked  from  tactical  units  for  the  test  duration, 
a.  M1A2  Test 

Training  and  familiarization  of  the  M1A2  test  crews  were  conducted  on  the  same 
ranges  and  maneuver  areas  as  the  OT&E  test  trials.  Decisions,  response  times,  and  accuracy  of 
weapons  firing  were  considered  superior  to  what  could  be  expected  in  a  combat  environment 
because  of  familiarity  with  the  test  ranges.  Many  of  the  battle  position  were  pre-designated  and 
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maneuver  was  artificially  controlled  to  meet  range  requirements.  Pre-testing  training  and  range 
control  procedures  contribute  in  making  Ml  A2  test  participants  atypically  qualified. 

b.  AH-64D  Apache 

Because  of  the  complex  nature  of  the  AH-64D,  the  operators  were  selected  from 
the  same  operational  unit,  A/2-229th.  (Stanfield,  March  1997)  Using  all  of  the  test  participants 
from  an  single  unit  could  bias  test  data.  For  example,  of  the  sixteen  pilots  for  the  operational  test, 
the  majority  were  senior  chief  warrant  officers  (CW3  or  CW4)  who  have  worked  together  over 
long  periods  of  time.  Their  pilot  skills  were  considered  to  be  atypically  high  when  compared 
with  less  experienced  pilots  prevalent  in  the  operational  units.  Additionally,  ranges  at  Hunter 
Liggett  required  the  use  of  common  battle  positions  for  testing.  Again,  pre-test  training  and 
familiarization  with  range  procedures  will  enhance  the  crews  ability  to  quickly  and  effectively 
engage  targets  from  designated  battle  positions. 

The  Distributive  Interactive  Simulation  (DIS)  network  provides  an  opportunity  to 
address  the  atypical  crew  problem.  Test  participants  selected  over  a  wide  geographic  area  can  be 
linked  together  through  DIS  and  participate  in  OT&E  testing  without  leaving  their  home  station. 
The  tendency  to  select  atypical  users  at  home  station  would  be  reduced  because  participants 
remain  anonymous  and  are  not  representing  the  unit  as  individuals.  The  same  techniques  could 
be  used  to  train  test  participants  without  familiarizing  them  with  the  limited  test  ranges  and 
maneuver  areas  used  for  live  test  events.  Data  collected  in  this  manner  would  enhance  reliability 
and  validity,  thus  improving  the  evaluation. 
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c.  Test  Data  Reliability  and  Validity  Summary 

Data  reliability  and  validity  are  affected  by  the  OT&E  test  scope,  realism 
achieved,  and  the  degree  to  which  test  participants  represent  typical  users.  Operational 
assessments  of  combat  effectiveness  and  suitability  are  directly  affected  by  data  validity  and 
reliability.  Integrating  M&S  into  the  OT&E  process  through  the  use  of  model-test-model  (MTM), 
virtual,  constructive,  and  Distributed  Interactive  Simulations  (DIS)  enhances  data  reliability  and 
validity,  and  therefore,  weapon  system  evaluations  become  more  accurate. 

The  uniform  quality  of  M&S-generated  data  appears  to  enhance  the  reliability  and 
validity  of  OT&E  data.  M&S-generated  data  are  less  susceptible  to  bias  because  the  variables 
are  controlled  by  design.  Additionally,  testing  can  be  replicated  for  reliability  and  validity. 

Fully  verified,  validated  and  accredited  M&S  applications  specifically  designed  for  data 
collection  produce  uniform  quality  data  which  should  enhance  the  evaluation  process. 

D.  SUMMARY 

Integration  of  advanced  M&S  techniques,  technologies,  and  applications  into  OT&E  can 
provide  a  means  of  addressing  operational  test  issues  beyond  the  scope  of  resource-constrained 
live  testing.  M&S  provides  a  vehicle  for  addressing  critical  issues  that  are  impossible  or 
impractical  to  represent  in  a  live  environment.  Operational  realism  and  combat  stress  may  be 
enhanced  through  M&S  application. 

Integrating  M&S  does  not  replace  or  eliminate  live  operational  testing,  but  rather 
augments  and  focuses  live  testing.  The  adage  "you  don’t  use  M&S  to  identify  what  is  exactly 
right,  you  use  it  to  eliminate  what  is  exactly  wrong  "  applies  to  OT&E  test  design.  Integrating 
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M&S  into  OT&E  design  reduces  variability  by  examining  critical  issues  across  the  operational 
spectrum  replicated  on  a  simulation. 


60 


VI .  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  OVERVIEW 

The  OT&E  process  has  a  major  impact  on  a  weapon  systems’  critical  acquisition 
decisions.  The  importance  of  OT&E  in  the  acquisition  process  is  underscored  by  Congressional 
interest  in  the  OT&E  process  and  it’s  results.  OT&E  assessments  are  reported  directly  to 
Congress  from  independent  OT&E  sources.  From  early  operational  assessments  through  the 
formal  operational  evaluation,  the  insights  to  a  weapon  systems'  characteristics  are  based  on 
operational  tests.  The  accuracy  of  OT&E  in  predicting  how  weapon  systems  will  operate  in  a 
combat  environment  directly  affects  future  combat  implementation,  and  more  importantly,  the 
survival  of  the  combat  troops  who  depend  upon  those  weapon  systems, 

M&S  integration  into  the  OT&E  process  has  traditionally  been  limited.  In  the  past,  M&S 
maturity  was  limited  to  physical  models  and  mathematical  simulation.  Advancements  in 
software,  hardware,  and  linked  communications  networks  have  resulted  in  explosive  M&S 
growth  and  maturity.  However,  the  OT&E  community  is  slow  to  integrate  advanced  M&S  due  to 
the  desire  to  protect  their  independent  status  and  the  perceived  negative  connotation  associated 
with  anything  "simulated"  in  a  live  test  environment. 

This  thesis  focused  on  the  potential  opportunities  to  integrate  M&S  concepts  and 
applications  into  the  Army  OT&E  process.  Recurring  OT&E  concerns  were  examined  and 
possible  M&S  based  enhancements  and  solutions  were  identified.  Increased  M&S  integration  in 
the  weapons  testing  will  enhance  the  OT&E  process. 


61 


B.  GENERAL  CONCLUSIONS 


Realistic  operational  combat  effects  on  personnel  and  equipment  cannot  be  wholly 
replicated  through  constructive  simulation  or  virtual  simulation  in  operational  testing.  Live 
testing  in  a  replicated  combat  environment  provides  the  most  realistic  operational  test,  however, 
it  is  the  most  expensive  OT&E  method.  A  coordinated  approach  to  operational  testing, 
integrating  live  tests  with  models  and  simulation  M&S,  provides  a  realistic  environment  for 
effective  OT&E.  M&S  cannot,  and  should  not  be  used  in  place  of  necessary  OT&E  live  testing. 
However,  M&S  applications  offer  methods  of  augmenting  and  enhancing  the  OT&E  process  that 
merit  consideration. 

The  tremendous  costs  associated  with  conducting  comprehensive  live  testing  will  limit  its 
use  in  the  budget  restrained  future.  Army  budgets  are  projected  to  continue  the  present 
downward  trend,  which  will  not  improve  the  prospects  for  expanding  OT&E  live  testing.  The 
requirement  to  create  holistic,  representative  operational  environments  conflicts  with  the  realities 
of  the  reduced  current  budget  and  projected  defense  funding  authorizations.  M&S  applications 
can  bridge  the  funding  gap,  assisting  both  the  PM  and  operational  testers  in  providing  good 
equipment  for  the  user. 

Interviews  and  research  indicate  there  are  significant  problems  and  concerns  associated 
with  the  Army  OT&E  process.  Some  believe  that  failures  during  OT&E  are  management 
failures,  rather  than  learning  opportunities.  Many  focus  on  the  inability  of  limited  OT&E  to 
adequately  assess  the  combat  effectiveness  and  suitability  of  weapon  systems,  as  required  by 
law.  Whatever  the  reason  for  limited  test,  it  can  lead  to  the  fielding  of  weapon  systems  whose 
operational  effectiveness  and  suitability  have  not  been  fully  demonstrated.  Inadequate 
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operational  testing  tends  to  yield  more  favorable  assessments  than  are  likely  to  be  found  when 
the  systems  are  employed  in  combat. 

The  integration  of  M&S  can  provide  some  solutions  to  recurring  OT&E  problems  and 
issues.  Integrating  M&S  into  the  OT&E  process  can  provide  valuable  insights  into  the 
operational  environment  beyond  the  scope  of  a  resource  constrained  OT&E  process.  However, 
M&S  cannot  fully  represent  the  personnel  stresses  associated  with  combat  environments. 
Additionally,  M&S  cannot  replicate  the  infinite  number  of  variables  present  in  live 
environments. 

C.  SPECIFIC  CONCLUSIONS 

1.  Test  Design 

Interviews  with  PM  and  OT&E  professionals  indicated  that  during  OT&E  it  is 
impossible  or  impractical  to  portray  the  full  range  of  force  structures.  Demonstrating  a  weapon 
system’s  integration  with  friendly  forces  and  interoperation  with  potential  threat  capabilities  is 
difficult  for  even  the  most  basic  system.  This  is  largely  due  to  the  fact  that  current  methods  for 
planning  OT&E  and  validating  that  plan  are  not  efficient  or  effective. 

Integrating  M&S  into  the  OT&E  test  design  process  can  assist  the  PM  and  test  planners  in 
addressing  the  problems  and  issues  associated  with  test  design.  Constructive  simulations  can 
provide  a  means  of  examining  the  candidate  weapon  system's  entire  operational  spectrum. 

Force  structures,  interoperating  systems,  and  threat  capabilities  can  be  represented  in  the 
constructive  environment.  Integrating  live  test  events  with  constructive  simulations,  such  as 
Model-Test-Model  applications,  can  identify  scenarios  and  environments  required  to  address 
critical  testing  issues. 
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2. 


Resource  Constraints 


Resource  constraints  limit  the  scope  of  live  OT&E  testing.  The  operational  test  often 
shrinks  as  resources  are  constrained.  Resource  constraints  limit  the  degree  to  which  realism  is 
replicated  in  a  live  test  environment.  These  constraints  increase  the  risk  that  the 
OT&E  will  neither  accurately  nor  adequately  address  the  combat  environment  that 
potential  weapon  systems  might  experience. 

Integrating  M&S  into  OT&E  testing  can  reduce  the  impact  created  by  resource 
constraints.  Constructive  and  virtual  simulations  can  provide  methods  to  address  operational 
issues  beyond  the  scope  of  resource  constrained  live  testing.  M&S  applications  are  more  cost 
effective  than  live  test  events  and,  therefore,  are  less  susceptible  to  the  negative  effects  resulting 
from  resource  constraints.  Virtual  simulations  accommodate  actual  users  in  an  unconstrained 
environment.  This  provides  critical  user  responses  in  environments  with  realistic  threat 
capabilities,  interoperability  with  other  battlefield  operating  systems,  and  force  structures 
impossible  to  replicate  in  a  live  environment. 

3.  Safety  and  Environmental  Concerns 

In  today’s  testing  environment,  safety  and  environmental  issues  increasingly  limit  the 
ability  to  fully  consider  critical  issues  in  a  live  OT&E.  New  technology  advancements  including 
lasers,  depleted  uranium  projectiles,  and  other  potentially  hazardous  systems  cannot  be  addressed 
in  an  unrestricted,  live  operational  test.  Systems  that  produce  environmentally  hazardous  by¬ 
products  are  restricted  to  very  controlled  scenarios,  not  representative  of  a  combat  environment. 
Additionally,  peacetime  safety  requirements,  significantly  more  restrictive  than  those  expected 
in  a  combat  environment,  negatively  impact  OT&E  assessments. 
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Integrating  M&S  into  the  OT&E  process  can  provide  solutions  for  data  collection  when 
scenarios  are  restricted  by  safety  or  environmental  concerns.  M&S  can  provide  a  means  to 
address  critical  issues  that  are  too  hazardous  or  environmentally  polluting  to  conduct  in  a  live 
test.  In  extreme  situations,  M&S  may  provide  the  only  OT&E  data  obtainable  for  OT&E 
evaluation. 

4.  Data  Reliability  and  Validity 

Limited  operational  testing  creates  data  reliability  and  validity  problems  and  issues.  Data 
collected  from  limited  OT&E  must  be  extrapolated  to  the  operational  spectrum  not  addressed 
within  the  scope  of  live  OT&E.  As  the  OT&E  test  scope  becomes  smaller  and  more  constrained, 
data  reliability  and  validity  risks  increase.  The  myriad  of  variables  present  in  a  live  OT&E 
environment  invites  questionable  interpretation  of  the  results.  False  causes  may  be  assigned  to 
some  undesirable  events  that  eliminate,  obscure,  or  reduce  the  certainty  of  the  failure. 

OT&E  test  participants  are  often  not  representative  of  the  typical  operational  user.  Data 
reliability  and  validly  suffer  when  "golden  crews"  (personnel  who  are  atypically  high  in  aptitude 
and  attitude)  are  used  to  represent  the  "typical"  user.  Training  test  participants  on  the  same 
ranges  and  maneuver  areas  used  for  OT&E  test  trials  creates  unrealistic  responses.  This 
familiarity  improves  response  times,  anticipation,  and  weapon  accuracy. 

Integrating  M&S  into  the  OT&E  process  can  assist  in  addressing  data  reliability  and 
validity  related  problems  and  issues.  M&S  applications  can  provide  a  means  to  augment  data 
required  for  quantitative  evaluation.  Data  generated  on  M&S  applications  have  controlled 
variables,  virtually  eliminating  the  opportunity  for  event  interpretation  and  manipulation.  DIS 
can  provide  an  effective  means  for  addressing  the  "golden  crew  "  problem.  Data  collection  and 
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test  participant  training  via  DIS  preserve  the  actions,  reactions,  and  decisions  of  representative 
operational  users. 

D.  RECOMMENDATIONS 

The  following  recommendations,  based  on  weapon  system  analysis,  should  enable  M&S 
to  enhance  and  improve  the  Army  OT&E  process: 

1.  The  Army  should  continue  to  expand  existing  M&S  efforts  in  requirements 
generation,  data  augmentation,  and  OT&E  testing.  Cooperative  M&S  efforts  between  OPTEC 
and  TRADOC  will  enhance  the  OT&E  process  through  early  and  accurate  critical  issue 
identification  and  test  design  validation.  As  a  result,  OT&E  test  data  will  be  improved  and 
combat  effectiveness  and  suitability  assessments  will  be  more  accurate. 

2.  M&S  applications  that  have  the  fidelity  to  integrate  candidate  system  capabilities 
and  parameters  for  OT&E  should  be  developed.  The  M&S  applications  developed  should 
portray  existing  threat  and  friendly  force  structures  from  small  unit  through  theater  level 
environments.  These  applications  should  include  constructive  and  virtual  simulations  that  can 
examine  the  impacts  on  threat  capability,  battlefield  interoperability,  and  logistics  supportability 
issues.  Accurate  simulations  provide  methods  for  examining  the  operational  spectmm  beyond  the 
live  test  scope.  Extrapolation  of  live  test  data  via  M&S  can  enhance  data  and  evaluation 
accuracy. 

3.  Verification,  Validation  and  Accreditation  (W&A)  requirements  for  previously 
validated  applications  should  be  limited  to  the  candidate  systems’ capabilities  and  parameters. 
The  W&A  process  for  M&S  applications  is  time  consuming  and  expensive.  M&S  applications 
with  pre-approved  VV&A  accreditation  could  enhance  M&S  integration  into  the  OT&E  process. 
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4.  Equipment  simulators  at  user  locations  should  be  linked  through  DIS.  Virtual 
simulations,  linked  through  the  DIS  network,  would  integrate  a  wide  variety  of  typical  users  for 
operational  assessments.  Test  participant  training  at  home  station  through  DIS  could  improve 
OT&E  data  validity  during  live  test  trials. 

E.  AREAS  FOR  FURTHER  RESEARCH 

Further  research  is  recommended  in  the  following  areas; 

1 .  Research  should  be  conducted  to  determine  the  differences  between  test  participant 
actions,  reactions,  and  decisions  in  live  versus  simulated  environments.  The  results  of  this 
research  should  help  establish  the  degree  to  which  M&S  generated  reaction  and  decision  data 
correlate  with  live  operational  testing. 

2.  The  integration  of  high  fidelity  M«&S  applications  should  be  analyzed.  Candidate 
system  employment  via  high  fidelity  simulations  could  provide  effectiveness  and  suitability 
assessments  while  varying  doctrine  and  tactics.  Additionally,  threat  forces  controlled  through 
high  fidelity  simulations  could  provide  probable  threat  adjustments  to  the  employment  of 
candidate  weapon  systems. 

3.  The  compatibility  of  current  and  future  weapon  systems  training  simulation  devices 
with  DIS  and  other  M&S  architectures  should  be  investigated.  Dual  use  of  training  simulation 
devices  for  training  and  OT&E  testing  via  virtual  and  distributed  simulations  could  enhance  the 
M&S  integration  into  the  OT&E  process. 
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APPENDIX 


ADS-  Advanced  Distributed  Simulations 

AMC-  Army  Material  Command 

AMIP-  Army  Model  Improvement  Program 

AMSAA-  Army  Materid  Systems  Analysis  Activity 

AMSMO-  Army  Modeling  and  Simulation  Management  Office 

AR-  Army  Regulation 


BDS-  Battlefield  Distributed  Simulation 

BDS-D-  Battlefield  Distributed  Simulation-Developmental 


C&C-  Command  and  Control 

C4I-  Command,  Control,  Communications,  Computers,  and  Intelligence 

CEP-  Concept  Evaluation  Program 

CITV-  Commander’s  Independent  Thermal  Viewer 

COI-  Critical  Operational  Issue 

CW3-  Chief  Warrant  Officer  -  3 

CW4-  Chief  Warrant  Officer  -  4 


DA  PAM-  Department  of  the  Army  Pamphlet 

DARPA-  Defense  Advanced  Research  Project  Agency 

DIS-  Distributed  Interactive  Simulation 

DMSO-  Defense  Modeling  and  Simulation  Office 

DoD-  Department  of  Defense 

DOT&E-  Director  of  Operational  Test  and  Evaluation 

DSB-  Defense  Science  Board 

DSI-  Defense  Simulation  Internet 

DT&E-  Developmental  Test  and  Evaluation 


EMD-  Engineering  and  Manufacturing  Development 
EUTE-  Early  User  Test  and  Evaluation 


FDTE-  Force  Development  Testing  and  Experimentation 
FFRDC-  Federally  Funded  Research  and  Development  Center 
FOTE-  Follow-on  Operational  Test  and  Evaluation 
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HW/SWIL-  Hardware/Software-in-the-loop 


lOTE-  Initial  Operational  Test  and  Evaluation 
IPPD-  Integrated  Product  and  Process  Development 
IVIS-  Intervehicular  Information  System 


JT&E-  Joint  Test  and  Evaluation 


LAM-  Louisiana  Maneuvers 
LRJP-  Low  Rate  Initial  Production 


M&S-  Modeling  and  Simulation 

MAIS-  Major  Automated  Information  systems 

MDAPS-  Major  Defense  Acquisition  Programs 

MNS-  Mission  Needs  Statement 

MOE-  Measure  of  Effectiveness 

MOO-  Measures  of  Outcome 

MOP-  Measure  of  Performance 

MOS-  Military  Occupational  Skill 

MTM-  Model-Test-Model 


OEC-  Operational  Evaluation  Command 
OPEVAL-  Operational  Evaluation 
OPTEC-  Operational  Test  and  Evaluation  Command 
ORD-  Operational  Requirements  Document 
OT&E-  Operational  Test  and  Evaluation 


PDRR-  Program  Definition  and  Risk  Reduction 

PDU-  Protocol  Data  Unit 

Ph-  Probability  of  Hit 

Pk-  Probability  of  Kill 

PM-  Program  Manager 

PPBS-  Planning,  Programming,  and  Budgeting 


R&D-  Research  and  Development 
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SAFOR-  Semi-automated  Forces 

SMNET-  Simulation  Network 

SIMTECH-  Simulation  Technology  Program 

STRICOM-  Simulation,  Training  and  Instrumentation  Command 


T&E-  Test  and  Evaluation 

TEC-  TEXCOM  Experimentation  Center 

TEMP-  Test  and  Evaluation  Master  Plan 

TEXCOM-  Test  and  Experimentation  Command 

TMDE-  Test  Measurement  Diagnostic  Equipment 

TRAC-  Training  and  Doctrine  Command  Analysis  Center 

TRADOC-  Training  and  Doctrine  Command 


W&A-  Verification,  Validation,  and  Accreditation 


WBS-  Work  Breakdown  Structure 
WSMR-  White  Sands  Missile  Range 
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